Papers

If you have ever played around with Large Language Models (LLMs) like GPT-4 or Llama, you have likely encountered In-Context Learning (ICL). It is the fascinating ability of these models to learn a new task simply by seeing a few examples in the prompt, without any gradient updates or weight changes. For instance, if you want a model to classify movie reviews, you might provide three examples of reviews and their sentiment (Positive/Negative) before asking it to classify a fourth one. This process seems magical and, crucially, it seems “free” compared to fine-tuning a model. ...

Introduction The landscape of sequence modeling is shifting. For years, the Transformer architecture has reigned supreme, driving the revolution in Large Language Models (LLMs). However, a new contender has emerged: State Space Models (SSMs), most notably the Mamba architecture. Mamba has generated significant excitement because it solves the Transformer’s biggest bottleneck: the quadratic computational cost of attention. Mamba scales linearly with sequence length, making it a potential “Transformer killer” for processing massive contexts. However, scaling Mamba to billions of parameters still presents a massive computational challenge. To deploy these models in real-world applications, we need to make them more efficient. ...

The Pruning Paradox: Why Minimizing Error in LLMs Can Backfire Large Language Models (LLMs) like LLaMA and GPT have revolutionized artificial intelligence, but they come with a heavy price tag: computational cost. Running these massive models requires significant memory and energy, creating a barrier to entry for many researchers and developers. To solve this, the field has turned to Neural Network Pruning—the art of removing parameters (weights) from a model to make it smaller and faster without losing too much intelligence. The standard approach is to treat pruning as a math problem: remove weights in a way that minimizes the difference between the original “dense” model and the new “sparse” model. This difference is called reconstruction error. ...

Imagine you are sitting in a room with a window open. A friend walks in, shivers slightly, and says, “It’s chilly in here.” If you are a literal thinker, you might simply agree: “Yes, the temperature is low.” But if you have social-pragmatic awareness, you understand the implicature: your friend wants you to close the window. This gap between literal meaning and intended meaning is the domain of Pragmatics. For humans, navigating these social nuances—implicatures, irony, humor, and metaphors—is intuitive. For Large Language Models (LLMs), it is notoriously difficult. While LLMs have mastered syntax and semantics, they often struggle to grasp the “unsaid” rules of human interaction. ...

Introduction: The Motorcycle Problem Imagine showing an AI model a picture of a person riding a motorcycle. You ask the model to describe what it sees. It replies: “A man riding a motorcycle.” Now, imagine that the rider is actually a woman. Why did the AI get it wrong? The answer lies in spurious correlations. In the vast datasets used to train these models, motorcycles appear significantly more often with men than with women. The model stops looking at the person and starts relying on the context: if there is a motorcycle, the model bets it’s a man. ...

Introduction: The “Coffee in Box Z” Problem Imagine you are given a logic puzzle: “The coffee is in Box Z, the stone is in Box M, the map is in Box H. What does Box Z contain?” For a human, this is trivial. You scan the sentence, find “Box Z,” look at what is associated with it (“coffee”), and give the answer. In cognitive science and linguistics, this process is known as binding. You are binding an entity (Box Z) to an attribute (coffee). ...

Imagine you are cooking with a robot assistant. You ask it to “pass the large bowl.” The robot reaches for a colander. You immediately say, “No, the ceramic one on the left.” The robot pauses, processes your correction, and successfully hands you the mixing bowl. This interaction seems trivial for humans. We constantly negotiate meaning in conversation. If we misunderstand something, we fix it and move on. However, for Artificial Intelligence—specifically Vision-Language Models (VLMs)—this process is incredibly difficult. Most current AI benchmarks focus on getting things right the first time based on a single instruction. But what happens when the AI gets it wrong? Can it recover? ...

Introduction In the world of machine learning, the mantra “data is fuel” has become a cliché, but it remains fundamentally true. The characteristics of a training dataset—its quality, diversity, and hidden biases—dictate the capabilities of the final model. However, analyzing this “fuel” is notoriously difficult. Currently, if a data scientist wants to understand their training data, they often look at attributes like “difficulty” (how hard is this sample to learn?) or “noisiness.” While useful, these metrics usually focus on individual instances in isolation. They fail to answer broader questions: How similar is Dataset A to Dataset B? Does this specific subset of 100 examples represent the knowledge of the entire dataset? ...

Introduction In the rapidly evolving landscape of Large Language Models (LLMs), generating text is only half the battle. The other half—and arguably the harder half—is evaluating that text. How do we know if a response is harmful, helpful, fluent, or consistent? Traditionally, we relied on metrics like BLEU or ROUGE, which simply count word overlaps between a model’s output and a human reference. But these metrics are rigid; they fail to capture nuance or semantic meaning. Recently, the industry has shifted toward “LLM-as-a-Judge,” where we ask a powerful model like GPT-4 to score a response. While effective, this approach is incredibly expensive, slow, and relies heavily on the model’s ability to articulate a critique. ...

Imagine you are asking a digital assistant to book a hotel. You specify “The Grand Budapest” in the “East” district. The bot replies confidently, “I have booked a room at The Grand Budapest.” But when you ask for the address, it gives you the location of a different hotel, simply because that other hotel is also in the East district and has a similar price range. In the world of Natural Language Processing (NLP), this is a classic case of a Task-Oriented Dialogue (TOD) system failing due to “distractive attributes.” The system retrieved the wrong knowledge entity because it looked deceptively similar to the right one. ...

Every researcher knows the feeling. You have a brilliant idea, you’ve run the experiments, and you’ve drafted the core methodology. Then, you hit the wall: the Related Work Section (RWS). To write a good RWS, you cannot simply list papers that sound similar to yours. You must craft a coherent narrative. You have to explain the history of the problem, group existing solutions by their approach, point out their flaws, and seamlessly transition into how your work fills the gap. It is a task that requires deep domain expertise, high-level synthesis skills, and the time to read hundreds of papers. ...

Imagine you are texting a friend for dinner recommendations. They tell you, “I absolutely hate spicy food; I can’t handle it at all.” You agree to go to a mild Italian place. Then, five minutes later, they text, “Actually, let’s get Indian, I eat spicy curry every single day.” You would probably be confused. You might scroll up to check if you misread the first message. You might ask them, “Wait, didn’t you just say you hate spice?” ...

Large Language Models (LLMs) like GPT-4 and Llama have revolutionized how we interact with text. They can write poetry, summarize emails, and even code. However, when you ask an LLM to perform a task that requires analyzing a complex, structured document—like an academic paper with dozens of citations—and assign it a specific numerical rating (such as a “disruption score”), the model often falters. The struggle stems from two main issues: structure and precision. First, standard LLMs read text linearly, but real-world documents are often hierarchical (trees of information). Second, LLMs are probabilistic text generators, not calculators; they struggle to output precise, continuous numerical values directly. ...

In the rapidly evolving landscape of Large Language Models (LLMs), we have moved beyond simple Q&A sessions. Users now engage in complex, multi-round conversations to refine code, write stories, or analyze data. Even more significantly, OpenAI’s introduction of “Custom GPTs” allows users to upload private data or have preparatory conversations to “prime” a bot for specific tasks. Ideally, these interactions are private. We assume that the context of our conversation—the “state” of the chat—is invisible to the outside world. ...

Imagine you are watching a movie, but instead of seeing the whole film, you are shown random, five-second clips in a shuffled order. In one clip, a character points to their left and laughs. In the next, they form a specific handshape and move it rapidly through the air. If you were asked to translate exactly what those actions meant, could you do it? Likely not. Without knowing who is standing to the left, or what object was established in the scene prior, you are guessing. ...

Large Language Models (LLMs) suffer from a critical limitation: they are frozen in time. Once trained, their knowledge is static. If the President of the United States changes, or if a new scientific discovery corrects a previous theory, the model remains ignorant until it undergoes expensive retraining or fine-tuning. To solve this, researchers developed Model Editing—techniques to surgical update specific facts inside a model without retraining the whole network. One of the most popular methods is ROME (Rank-One Model Editing). It has been hailed as a breakthrough for its ability to locate and edit specific factual associations. ...

When we interact with Large Language Models (LLMs) like GPT-4 or LLaMA, it is easy to be seduced by their apparent intelligence. You ask a complex multi-step question, and the model produces a coherent, logical answer. It feels like thinking. But under the hood, is the model actually reasoning? Or is it simply engaging in a sophisticated form of pattern matching, stitching together cues from your prompt to hallucinate a logical structure? ...

Introduction In the rapidly evolving landscape of Large Language Models (LLMs), a new “reasoning strategy” seems to drop every week. We’ve moved far beyond simple prompts. We now have agents that debate each other, algorithms that build “trees” of thoughts, and systems that reflect on their own errors to self-correct. Papers introducing these methods often show impressive tables where their new, complex strategy dominates the leaderboard, leaving simple prompting techniques in the dust. But there is a catch—a hidden variable that is often overlooked in academic comparisons: Compute Budget. ...

Imagine you are looking at a photograph of a room you’ve never visited. Someone asks, “Will that couch fit through the doorway?” Even though you don’t have a tape measure, you can probably make a very good guess. You intuitively know that a standard door is about 80 inches high, and using that mental “ruler,” you estimate the size of the couch. This ability to use context clues to measure the world is second nature to humans. ...

If you are studying software security or machine learning, you have likely noticed the explosion of interest in Large Language Models (LLMs). We know LLMs can write code, explain algorithms, and even translate languages. But can they act as security auditors? Can they look at a piece of code and tell you, “Hey, there’s a dangerous SQL injection right here”? The short answer is yes, but with major caveats. While there has been significant research into using Deep Learning for vulnerability detection in languages like C and C++, the web’s most dominant language—PHP—has been largely left behind. This is a critical gap. PHP powers nearly 80% of the top ten million websites, including giants like WordPress and Wikipedia. ...