GPT-3: The Dawn of Few-Shot Learning

The Fine-Tuning Treadmill: A Problem of Scale

For years, the dominant paradigm in Natural Language Processing (NLP) has been a two-step dance. First, pre-train a massive, general-purpose language model on a vast ocean of text data. These models, such as BERT or RoBERTa, learn intricate patterns of language—grammar, facts, reasoning abilities, and even some biases.

The second step is to take this powerful but general model and specialize it for a specific task through fine-tuning.

Want a sentiment classifier? You need a large, labeled dataset of positive and negative reviews. A question-answering bot? Thousands of question–answer pairs. For every new task, you build a new dataset and a new fine-tuned model. This “pre-train then fine-tune” approach has been remarkably effective, pushing the state-of-the-art in countless benchmarks.

But this method has a fundamental limitation: it demands large amounts of task-specific labeled data, which is often expensive and time-consuming to create. It also feels unlike human learning—we don’t need thousands of examples to understand an instruction like “translate this sentence to French.”

This is the world that the 2020 paper, “Language Models are Few-Shot Learners”, dramatically changed. The authors from OpenAI posed a profound question:

What if we simply kept scaling? What if we built a language model so large and trained it on so much diverse data that it could perform tasks with only a handful of examples—or even none—without fine-tuning?

The answer led to GPT-3—a colossal 175-billion-parameter model with an emergent ability for in-context learning. This article unpacks how GPT-3 works, what makes it a few-shot learner, and the remarkable—sometimes concerning—implications of its capabilities.

A New Way to Learn: From Fine-Tuning to Few-Shot

Before diving into GPT-3, let’s define the four approaches the paper describes, spanning from data-heavy to data-scarce:

1. Fine-Tuning (FT): Update a pre-trained model’s weights by training on thousands of labeled examples for a target task. This often yields the strongest performance, but is the most data-hungry.

2. Few-Shot (FS): At inference time, provide the model with a prompt that includes a handful of demonstrations of the task.
   Example for English-to-French translation:

   english: sea otter => french: loutre de mer
   english: peppermint => french: menthe poivrée
   english: cheese => french:

   Here, K is small—typically 10–100 examples fit within the 2048-token context window. No weights are updated; the model’s “learning” happens purely through the prompt context.

3. One-Shot (1S): Same as few-shot, but with only a single example (K=1).

4. Zero-Shot (0S): Provide an instruction in natural language with no examples:

   Translate English to French:
   cheese =>

   The model relies entirely on its learned knowledge of what “translate” means.

This capability—in-context learning—emerges from training a model at massive scale to predict the next word. The authors argue the model uses its large context window as a kind of short-term memory, learning patterns on-the-fly without gradient updates.

Building the Beast: The GPT-3 Architecture

GPT-3 is an autoregressive Transformer-based language model, predicting the next token from the sequence so far—like its predecessor GPT-2, but far bigger.

Scale

• 8 model sizes from 125M parameters up to 175B.
• GPT-3 is >100× larger than BERT-Large (340M), and >10× larger than the largest dense language model before it.

Training Data

A curated mix of ~500B tokens from:

• Filtered Common Crawl (massive public web snapshot).
• Expanded WebText dataset from GPT-2’s training.
• Two internet-based book corpora (Books1, Books2).
• English-language Wikipedia.

The data was deduplicated and mostly seen only once, leveraging prior scaling law findings that large models learn better from more unique data.

Figure 1: Key findings. Left: SuperGLUE performance rises with model size; few-shot (orange) gains most. Middle: More examples in context (K) steadily improve few-shot performance. Right: Across 42 benchmarks, accuracy improves with scale; few-shot consistently outperforms one- and zero-shot.

Putting GPT-3 to the Test

The central hypothesis: performance should improve smoothly with scale. As Figure 1 shows, all settings—zero-, one-, and few-shot—improve with model size, and the gap between zero- and few-shot widens for larger models. Bigger models are better in-context learners.

Cloze and Completion Tasks

The LAMBADA dataset tests predicting the last word of a paragraph. Few-shot GPT-3 achieved 86.4% accuracy, jumping far beyond the prior SOTA of 68%.

Table 1: On LAMBADA, few-shot GPT-3 obliterates previous SOTA. It also performs strongly on StoryCloze and HellaSwag, though not SOTA-level.

Question Answering

In closed-book QA (no retrieval), GPT-3 excelled. On TriviaQA, few-shot GPT-3 scored 71.2% accuracy, a new SOTA—outperforming fine-tuned retrieval-based systems.

Table 2: In closed-book QA, GPT-3 few-shot surpasses even open-domain systems for TriviaQA.

Some tasks revealed limits. On DROP, involving discrete numerical reasoning, GPT-3 outperformed fine-tuned BERT but lagged far behind humans and specialist models.

Table 3: GPT-3 is strong on conversational QA (CoQA) but weaker on discrete-reasoning datasets like DROP.

Machine Translation

Although 93% of training data was English, GPT-3 could translate. Few-shot GPT-3 set new SOTA for unsupervised translation into English from French, German, and Romanian. Translating out of English was weaker, reflecting its English-language bias.

Table 4: Few-shot GPT-3 shines translating into English, surpassing prior unsupervised work.

SuperGLUE Benchmark

SuperGLUE combines diverse, tough NLP tasks. GPT-3’s few-shot results were mixed—near SOTA on COPA and ReCoRD, competitive with BERT-Large on several others, but poor on WiC, which involves cross-sentence word meaning comparison.

Table 5: Strong results on COPA and ReCoRD; weak on WiC, CB, and RTE, especially for tasks requiring direct sentence comparison.

The Dark Side of Scale: Bias, Misuse, and Cost

The paper discusses not only achievements but also the broader risks.

Can You Spot the Bot?

Researchers tested human ability to distinguish GPT-3-generated news articles from human-written ones. For GPT-3 175B, humans averaged 52% accuracy—barely above chance. Larger models were harder to detect.

Figure 2: Bigger models produce text so human-like it’s barely distinguishable from real writing.

This raises misuse concerns—mass misinformation, spam, phishing, and automated abuse.

Internet-Scale Biases

Training data from the internet brings internet-scale bias. Preliminary analysis found:

• Gender: Occupations skew to stereotypical gender associations (e.g., “nurse” → female, “professor emeritus” → male).
• Race: Prompts about “Asian” led to consistently positive sentiment; “Black” skewed negative.
• Religion: Islam co-occurred more with terms like “terrorism” than other religions.

Table 6: Learned stereotypes, e.g., “terrorism” appearing more with Islam prompts, reflect biases in training data.

Environmental and Financial Cost

Training GPT-3 175B took several thousand petaflop/s-days—orders of magnitude beyond prior models.

Figure 3: Compute requirements have exploded; GPT-3 dwarfs predecessors like BERT and T5.

The authors note costs can be amortized by using one model across tasks without retraining, but the “bigger is better” trend challenges sustainability.

Conclusion: A Paradigm Shift

“Language Models are Few-Shot Learners” is about more than just scaling up—it redefines how we think about using language models. GPT-3 shows that with enough parameters, data, and compute, a single model can flexibly perform many tasks with little to no task-specific training.

This drastically reduces reliance on large labeled datasets, bringing us closer to adaptable general-purpose AI. Yet, the work also issues clear warnings: bias, potential misuse, and the massive cost of training such models are as important as their technical milestones.

GPT-3 marks the beginning of a new era—forcing us to ask not only what these models can do, but what they should do.