EMNLP 2024

The scale of Large Language Models (LLMs) is exploding. From GPT-4 to Llama, models are getting bigger, smarter, and—crucially—much more expensive to run. The primary culprit for this cost is the dense nature of these architectures: every time you ask a question, every single parameter in the model is activated to calculate the answer. Imagine a library where, to answer a single question, the librarian has to open and read every single book on the shelves. That is a dense model. ...

If you have ever tried to train a Large Language Model (LLM) from scratch, you know the pain. It requires massive computational resources, vast amounts of data, and a budget that usually only large tech giants possess. But here is a question that has puzzled researchers: if we already have excellent models fluent in English (like LLaMA), why do we burn millions of dollars training new models from scratch just to teach them a new language like Chinese? ...

Introduction Imagine you are trying to teach a computer how to handle complex customer service calls—for example, booking a multi-leg flight while simultaneously reserving a hotel and buying tickets for a local attraction. In the world of Artificial Intelligence, specifically Task-Oriented Dialogue (ToD) systems, this is a massive challenge. The standard approach is Reinforcement Learning (RL). The AI talks to a user simulator, tries to fulfill the request, and gets a “reward” (a positive signal) only if it completes the entire task perfectly. If it fails, it gets nothing or a penalty. This is known as the sparse reward problem. It is akin to trying to learn how to play a piano concerto by hitting random keys and only being told “good job” if you accidentally play the whole piece perfectly on the first try. ...

Introduction Imagine trying to explain the structure of an atom to someone who has never taken a physics class. You could recite a textbook definition about protons, neutrons, and electron shells. Or, you could say: “The atom is like a solar system. The nucleus is the sun in the center, and the electrons are planets orbiting around it.” For most learners, the second explanation—the analogy—is the one that clicks. Analogical reasoning is a cornerstone of human cognition. It allows us to map the familiar (the solar system) onto the unfamiliar (the atom), building a bridge to new understanding. ...

Introduction In the age of information overload, the ability to distinguish between a sound argument and a deceptive one is more critical than ever. We often rely on Large Language Models (LLMs) to summarize news, analyze debates, or verify facts. However, while LLMs are incredibly fluent in generating text, they frequently struggle with the nuance of logical reasoning. They can be easily swayed by arguments that sound coherent but are structurally flawed. ...

Large Language Models (LLMs) have revolutionized how we process information, acting as capable assistants for summarization, dialogue, and question answering. However, anyone who has used them extensively knows their Achilles’ heel: they don’t know everything. Their knowledge is frozen in time at the moment of training, and they can confidently “hallucinate” incorrect facts. To solve this, the AI community adopted Retrieval-Augmented Generation (RAG). The idea is simple: before the LLM answers, it searches a database (like Wikipedia), finds relevant documents, and uses that information to generate an answer. ...

Introduction In the current landscape of Natural Language Processing (NLP), the Transformer architecture reigns supreme. From ChatGPT to Llama, the mechanism of self-attention has unlocked incredible capabilities in generation and reasoning. However, this power comes with a significant computational cost. Attention scales quadratically with sequence length, and the Key-Value (KV) cache grows linearly, making the processing of massive contexts increasingly expensive for both training and deployment. This scaling bottleneck has reignited interest in efficient alternatives, specifically State Space Models (SSMs). Models like Mamba, S4, and Hawk promise the “holy grail” of sequence modeling: linear scaling and a fixed-size state that allows for constant-cost inference. In theory, they are the perfect solution for long-context applications. ...

Introduction The rate at which biomedical literature is published is staggering. Every day, thousands of new papers are released, detailing novel drug interactions, genetic discoveries, and disease mechanisms. For researchers and clinicians, keeping up with this flood of information is impossible. Yet, hidden within these unstructured texts are the keys to new cures and therapies. To manage this, we rely on Information Extraction (IE)—using AI to automatically parse text and convert it into structured databases. This typically involves two steps: Named Entity Recognition (NER) (identifying distinct items like “Aspirin” or “TP53”) and Relation Extraction (RE) (determining how they interact, e.g., “Aspirin inhibits TP53”). ...

Introduction In the age of social media, automated content moderation is not just a luxury; it is a necessity. Platforms rely on sophisticated AI models to filter out toxic speech, harassment, and hate speech to keep online communities safe. However, these guardians of digital safety have a hidden flaw: they often become prejudiced themselves. Imagine a scenario where a user types, “I am a proud gay man.” A toxic content classifier might flag this as “Toxic” or “Hate Speech.” Why? Not because the sentiment is hateful, but because the model has learned a spurious correlation between the word “gay” and toxicity during its training. This phenomenon is known as unintended bias or false positive bias. ...

Large Language Models (LLMs) like GPT-4 and Llama-2 have revolutionized how we interact with technology. They draft our emails, debug our code, and answer our most complex questions. However, these models are mirrors of the data they were trained on—data that reflects the internet, which unfortunately includes historical prejudices, stereotypes, and social biases. For researchers and developers, ensuring these models are fair is a massive priority. But there is a technical bottleneck: how do we actually measure fairness? Traditional methods rely on rigid templates or statistical probabilities that don’t reflect how we use AI today. We don’t use ChatGPT to fill in the blank of a sentence; we engage in open-ended conversation. ...

Have you ever tried to interrupt a voice assistant? It usually goes something like this: you ask a question, realize you made a mistake mid-sentence, but the AI ignores your correction and continues to process your first request. You have to wait for it to finish a long monologue, or frantically hit a “stop” button, before you can try again. This awkward dance happens because almost all current Large Language Models (LLMs) operate on a turn-based mechanism. You speak, the model waits for you to finish, it processes, and then it speaks. It is the digital equivalent of using a walkie-talkie (“Over and out”). ...

Introduction: The “Walkie-Talkie” Problem If you have ever conversed with a voice assistant like Alexa, Siri, or current iterations of ChatGPT Voice, you have experienced a “half-duplex” interaction. Much like using a walkie-talkie, the protocol is rigid: you speak, you stop, the machine detects silence, processes your request, and finally responds. This turn-based exchange is functional, but it is distinctly non-human. Real human conversation is “full-duplex.” It is a chaotic, synchronized dance. We interrupt each other to clarify points. We offer verbal “backchannels” (like “uh-huh,” “right,” or “yeah”) while the other person is still talking to signal engagement. We anticipate what the other person is about to say before they finish their sentence. ...

In the world of Machine Translation (MT), we have reached a fascinating tipping point. For decades, the goal of a translation system was to match human performance. Today, with the advent of Large Language Models (LLMs) like GPT-4, machine-generated translations often exceed the quality of human-written references. This creates a paradox in evaluation. Traditional metrics work by comparing the machine’s output (the “candidate”) to a human translation (the “reference”). If the reference is treated as the “Gold Standard,” how can a metric possibly reward a machine for writing something better? ...

In the high-stakes world of healthcare, Artificial Intelligence is rapidly becoming an indispensable tool. One of the most critical back-office tasks in medicine is medical coding—the translation of unstructured clinical text (like doctor’s notes) into standardized International Classification of Diseases (ICD) codes. These codes are vital for billing, epidemiology, and treatment tracking. While Large Language Models (LLMs) have shown incredible prowess in automating this task, they suffer from a significant “black box” problem. When an AI assigns a code for “postoperative wound infection,” a doctor needs to know why. If the model cannot explain its reasoning, it cannot be trusted in a clinical setting. ...

Introduction Imagine showing an AI a picture of a U.S. five-dollar bill. A standard computer vision model looks at the pixels and recognizes patterns: it sees paper, a face, and numbers. It can tell you “this is a banknote.” But what if you ask, “Who is the person in the portrait, and what specific historical event did he lead the country through?” To answer this, the model needs more than just visual pattern matching. It needs world knowledge. It needs to know that the face belongs to Abraham Lincoln, that Lincoln was the 16th U.S. President, and that he led the U.S. through the Civil War. Standard visual embeddings—the vector representations models use to “see”—often fail to capture this depth of instance-level knowledge. ...

Introduction In the world of Machine Translation (MT), we have witnessed a massive shift from heuristic-based evaluation metrics (like BLEU) to neural-based metrics (like COMET and MetricX). These newer models are significantly better at aligning with human judgments. However, they come with a “black box” problem. When a neural metric hands you a score—say, 0.86—what does that actually mean? Is it a perfect translation? Is it just “okay”? If another metric gives the same sentence a -1.49, how do you compare them? ...

Introduction In recent years, Multimodal Vision-Language Models (VLMs) like GPT-4V and Gemini have demonstrated an astonishing ability to interpret images. They can identify objects, read text within photos, and describe complex scenes. However, recognizing a “wedding” is one thing; understanding the specific rituals, clothing, and traditions associated with a wedding in rural India versus one in Ethiopia is a different challenge entirely. As digital interactions become increasingly global, AI models must move beyond general object recognition to grasp cultural values—the shared beliefs, rituals, and traditions that define human societies. ...

Imagine you are told that “Tweety is a bird.” Based on your general knowledge, you logically infer that “Tweety flies.” But a moment later, you receive a new piece of information: “Tweety is a penguin.” What happens in your brain? You immediately revise your belief. You retract the conclusion that Tweety flies, but you maintain the premise that he is a bird. You have just performed belief revision—the cognitive ability to update your understanding when new evidence contradicts or contextualizes what you previously thought was true. ...

Imagine you are having a terrible day. You turn to a friend to vent about your stress. In response, they give you a single-word reply: “Okay.” You feel unheard and indifferent. Now, imagine the opposite scenario. You share your problem, and that same friend responds with a ten-minute, breathless monologue, analyzing every micro-factor of your situation, citing historical precedents, and offering fifteen different solution paths simultaneously. You feel overwhelmed. Instead of feeling supported, you are now exhausted. ...

Introduction In the rapidly evolving world of Artificial Intelligence, Large Language Models (LLMs) have gained a reputation for being quick learners. Specifically, they excel at In-Context Learning (ICL). This is the ability to adapt to a new task simply by seeing a few examples in the prompt, without requiring any updates to the model’s weights. Imagine you want an AI to translate English slang into formal text. You don’t need to retrain it; you just provide a few pairs: “Gonna -> Going to” and “Wanna -> Want to”, and the model figures out the pattern for the next input. ...