Papers

How does the human mind handle time? It’s a question that feels both simple and impossibly complex. So much of what we do—from understanding a melody to catching a ball to having a conversation—depends on processing sequences of events as they unfold. Language, in particular, is a river of information flowing through time. The meaning of a sentence isn’t just in the words themselves, but in their order. “Dog bites man” is ordinary news; “Man bites dog” is a headline. ...

When designing deep neural networks, we usually focus on how data flows forward through the model. We stack layers, implement complex feature fusion mechanisms, and add attention modules to transform an input into the desired output. This traditional “data path” perspective has brought us powerful architectures like ResNet, DenseNet, and Transformers. But what if this forward-focused view is only half the story? What if the key to building more efficient and more powerful networks is to examine how information flows backward? ...

Large language models (LLMs) have shown astonishing capabilities: writing code, composing essays, and answering complex questions. Much of that success rests on few-shot learning—showing a model a few examples in the prompt and letting it generalize. But few-shot prompting has drawbacks: you need examples, and you often must engineer the prompt carefully. What if we could simply tell a model, in plain English, what we want it to do—and have it do it well without any example? That’s the core question of “Finetuned Language Models Are Zero-Shot Learners” (Google Research). The paper shows that a surprisingly simple trick—instruction tuning—turns large pretrained models into strong zero-shot learners. The instruction-tuned model, FLAN (Finetuned Language Net), improves zero-shot performance across many tasks and even beats GPT-3 (175B) zero-shot on most evaluated datasets. ...

The Fine-Tuning Treadmill: A Problem of Scale For years, the dominant paradigm in Natural Language Processing (NLP) has been a two-step dance. First, pre-train a massive, general-purpose language model on a vast ocean of text data. These models, such as BERT or RoBERTa, learn intricate patterns of language—grammar, facts, reasoning abilities, and even some biases. The second step is to take this powerful but general model and specialize it for a specific task through fine-tuning. ...

For decades, the idea of an AI that could write its own code has been a holy grail of computer science. We’ve seen glimpses of this future in science fiction, but in reality, teaching a machine the logic, creativity, and precision required for programming has been an immense challenge. When large language models (LLMs) like GPT-3 emerged, they revealed a surprising, albeit rudimentary, ability to generate simple code snippets from natural language prompts — even though they weren’t explicitly trained to code. ...

In the world of AI—and especially in Natural Language Processing (NLP)—the mantra for the past few years has been “bigger is better.” We’ve seen a parade of colossal language models like GPT-3, T5, and Megatron, each pushing the boundaries of size and performance. Scaling these models has unlocked incredible capabilities, from writing coherent essays to generating code. But it comes at a steep price: astronomical computational costs. Training these massive dense models, where every parameter is used for every single input, requires supercomputers and consumes enormous amounts of energy. ...

The world of Artificial Intelligence is in an arms race, but the weapons aren’t missiles—they’re parameters. From BERT (340 million) to GPT-2 (1.5 billion) and T5 (11 billion), we’ve seen a clear trend: bigger models tend to deliver better accuracy. But this relentless growth comes at a steep price—training these behemoths demands an astronomical amount of memory, far exceeding what a single GPU can handle. Consider this: even a modest 1.5-billion-parameter model, like GPT-2, requires more than 24 GB of memory just for training states when using standard methods. That already pushes the limits of a high-end 32 GB GPU—and that’s before you account for the activations and all the temporary data. So how can we possibly train models with tens, hundreds, or even a trillion parameters? ...

In the world of artificial intelligence, Large Language Models (LLMs) can seem like a form of modern alchemy. We mix massive datasets, gargantuan neural networks, and mind-boggling amounts of computation—and out comes something that can write poetry, debug code, and explain complex topics. But why does this work? And if we had ten times the resources, how much better could we make it? Is there a method to this madness, or are we just hoping for the best? ...

The recent explosion of progress in Natural Language Processing (NLP) largely rides on one lesson: pre-train large models on lots of text, then adapt them to specific tasks. Models like BERT, GPT-2, RoBERTa, and XLNet all lean on this transfer-learning paradigm, but they differ in architecture, pre-training objectives, and datasets — and those differences can be hard to disentangle. In “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer” the Google Brain team took a different tack. Instead of proposing just another tweak, they built a unified experimental playground and ran a massive, principled sweep of variables: architectures, unsupervised objectives, pre-training corpora, fine-tuning strategies, and scaling regimes. The result is both a thorough empirical guide and a family of state-of-the-art models called T5 (Text-to-Text Transfer Transformer). ...

In the world of AI today, models like ChatGPT seem almost magical. They can write code, compose poetry, and answer complex questions with remarkable fluency. But this revolution didn’t happen overnight—it was built on a series of foundational breakthroughs. One of the most crucial was a 2018 paper from OpenAI titled “Improving Language Understanding by Generative Pre-Training”. This paper introduced what we now call GPT-1, presenting a simple yet profoundly effective framework that changed the trajectory of Natural Language Processing (NLP). The core idea: first let a model learn the patterns of language from a massive amount of raw text, and then fine-tune that knowledge for specific tasks. ...

The world of Natural Language Processing (NLP) has entered an era of giant models. From GPT-2 to BERT and beyond, one trend is crystal clear: the bigger the model, the better the performance. These massive transformers can generate coherent articles, answer complex questions, and parse language with unprecedented nuance. But this capability comes at a steep engineering cost. These models have billions—and increasingly, trillions—of parameters. How can such colossal networks fit into the memory of a single GPU? ...

Deep learning models are notoriously data-hungry. The more high-quality, labeled data you can feed them, the better they perform. But what happens when you can’t just collect more data? You get creative. For years, the go-to technique has been data augmentation: taking your existing images and creating new, slightly modified versions—flipping them, rotating them, shifting colors—to expand your dataset for free. This approach works wonders. It teaches the model what features are truly important and which are just artifacts of a specific image. A cat is still a cat whether it’s on the left or right side of the frame. This concept of invariance—knowing which changes don’t alter the label—is key to building robust models. ...

Large Language Models (LLMs) have shown an extraordinary capacity to improve themselves through reinforcement learning (RL). By generating solutions, receiving feedback, and adjusting their strategy, they can learn to tackle complex problems such as advanced mathematical reasoning. This process hinges on a critical step: exploration—trying many different approaches, or “rollouts,” to discover what works. The catch? Exploration is computationally expensive. Generating thousands of possible solutions for thousands of different problems consumes massive amounts of GPU time. To keep costs manageable, current methods typically assign a small, fixed exploration budget to every problem—often 8 possible solutions per task. ...

Large Language Models (LLMs) are transforming software development. From autocompleting entire functions to answering complex, repository-level questions, these AI assistants are quickly becoming indispensable. But they have an Achilles’ heel: context length. When you ask an LLM to work with a large project, you often end up feeding it tens of thousands of lines of code. This long context creates a perfect storm of problems: Lost in the middle: Models can struggle to identify relevant pieces as important tokens get buried. Slow generation: Attention mechanisms scale quadratically, so long inputs cause latency to skyrocket. High costs: Commercial APIs charge by the token. Long contexts quickly run up the bill. For source code, this is particularly problematic. Unlike prose, code has intricate interdependencies. A function in one file might be essential to logic spread across dozens of others. Randomly chopping off text to fit a budget risks breaking compile-ready structure and losing critical constraints. ...

The world of 3D graphics is undergoing a revolution. For decades, creating photorealistic 3D scenes was the domain of skilled artists using complex software. But modern techniques like Neural Radiance Fields (NeRF) and, more recently, 3D Gaussian Splatting (3DGS) have profoundly changed the game. These methods can learn a stunningly accurate 3D representation of a scene from just a handful of 2D images, enabling applications from virtual reality and digital twins to advanced visual effects. ...

Introduction: Beyond Just Words Imagine searching for a specific chart hidden in hundreds of pages of financial reports, or trying to locate a product in a sprawling digital catalog using both its image and a brief description. In today’s increasingly multimedia world, documents are more than just text—they are rich ecosystems of words, images, layouts, charts, and tables. Traditional text-only search engines often fail to capture the meaning locked inside these visual elements, missing vital context. ...

Introduction: The Power of Thinking Before You Speak We’ve all heard the advice, “think before you speak.” It’s a core aspect of human intelligence—the ability to pause, reason through the consequences, and formulate a thoughtful response. Nobel laureate Daniel Kahneman described this reflective, deliberate process as System 2 thinking: the kind of mental effort that distinguishes a knee-jerk reaction from a reasoned argument. For much of their existence, Large Language Models (LLMs) have operated more like System 1 thinkers: remarkably fast, impressively fluent, but too often shallow in reasoning. Recent research has sought to change that by teaching models to “think” before answering, using a strategy called Reinforcement Learning with Verifiable Rewards (RLVR). In RLVR, a model generates a long chain of thought (CoT) before producing its answer, and earns a reward when the final answer can be automatically verified as correct. This works extremely well in math and code—where correctness is objective. If the math checks out or the code passes all the unit tests, the model gets rewarded. ...

Large Language Models (LLMs) like GPT-4 and Claude are incredible reasoning engines. You can ask them almost anything, and they’ll produce a coherent—often correct—answer. But they have an Achilles’ heel: their knowledge is internalized. It’s baked in during training, and once the training finishes, that knowledge becomes static. This means it can be outdated, incorrect, or simply missing, especially for specialized or rapidly changing domains. This leads to the infamous problem of hallucination, where an LLM confidently states something factually wrong. ...

Large Language Models (LLMs) have taken the world by storm, and their remarkable abilities stem from a deceptively simple principle: predict the next word. This approach, known as autoregressive generation or input-space reconstruction, has been the bedrock of models like GPT, Llama, and Gemma. But what if this cornerstone of LLM training is also a limitation? In computer vision, researchers have discovered that moving away from raw pixel reconstruction and instead training in a more abstract embedding space yields far superior results. A leading paradigm here is the Joint Embedding Predictive Architecture (JEPA), which encourages models to understand the essence of an image rather than memorize superficial details. ...

Large Language Models (LLMs) like GPT-4 and Llama-3 have taken the world by storm. They can write poetry, debug code, and even ace university exams. But ask one to perform a task that requires strict, step-by-step logical reasoning—like assembling a complex piece of furniture or planning a logistics route—and you might find the cracks in their armor. While LLMs are masters of language and general knowledge, they often stumble when faced with problems that demand formal, structured planning. They might propose impossible actions, overlook consequences of previous steps, or fail to detect when a goal has been met. ...