Large Language Models (LLMs) are famously knowledgeable. Ask them about the capital of France, the history of the Roman Empire, or the syntax of Python, and they will likely give you a correct answer. However, a lingering question in the field of AI safety and reliability is not just about what models know, but whether they know what they know.

More specifically: Does an LLM understand the scope of its own knowledge?

If a model knows three facts about a specific topic, does it know that it only knows three facts? Or will it recite the three facts and then hallucinate a fourth because it doesn’t realize it has run out of information? This concept—the awareness of the quantity and boundaries of one’s own knowledge—is a crucial component of intelligence. Without it, AI systems are prone to overconfidence, redundancy, and fabrication.

In the paper “Do Large Language Models Know How Much They Know?”, researchers from Mila, Meta FAIR, and Université de Montréal developed a novel benchmark to test this capability. Their findings shed light on the internal mechanisms of models like OPT, Pythia, and Flan-T5, revealing that “knowing how much you know” is an emergent ability that depends heavily on scale and architecture.

The Problem of Knowledge Scope

Current research often focuses on whether an LLM can answer a specific question (e.g., “Is Jupiter a planet?”). This is a binary assessment: the model either retrieves the fact or it doesn’t.

However, real-world knowledge is rarely singular. It is often a collection of facts dispersed across various documents. For an AI to be truly reliable, it needs to perform implicit knowledge retrieval—it must search its internal parameter space (its “brain”), retrieve multiple pieces of information, and, crucially, recognize when the search is complete.

If a model cannot quantify its expertise, it cannot be trusted to be exhaustive without being deceptive. The researchers set out to investigate this by asking models to enumerate everything they know about a specific topic—no more, no less.

Methodology: The Diary Benchmark

To test this, the researchers could not use public data like Wikipedia. If they asked GPT-4 to “list all facts about Barack Obama,” evaluating the answer would be impossible because we don’t know exactly which documents the model saw during training.

Instead, the researchers created a synthetic dataset consisting of diary entries written by fictitious individuals. This allowed for precise control over the “ground truth.”

The Setup

  1. Data Generation: The team generated thousands of diary entries for fictional characters (e.g., Tom, Alice, Bob).
  2. Variables:
  • Each character wrote a random number of entries (e.g., Alice wrote 3, Bob wrote 5).
  • Each entry contained random attributes (Location, Weather, Mood, Activity).
  1. Training: The models were fine-tuned on these documents, effectively memorizing the lives of these fictional people.

The Task

During the evaluation phase, the models were given a simple prompt: “Recall all of {Name}’s diary entries, in order.”

To succeed, the model must:

  1. Identify the individual.
  2. Retrieve every specific diary entry associated with that individual from its memory.
  3. Stop exactly when it runs out of real entries.

This diagram illustrates how Large Language Models (LLMs) can be trained on diary entries and then evaluated using retrieval tasks. The left side shows colored boxes representing diary entries for Tom, Bob, and Alice. The right side shows the Training phase (next-token prediction) and the Evaluation phase (recalling all entries for Alice).

As shown in Figure 1, if the model recalls Alice’s entries, it must generate exactly the entries she wrote. If Alice wrote three entries, and the model generates two, it has failed to recall the scope of its knowledge. If it generates four, it has hallucinated.

Experiments: Can Models Count Their Memories?

The researchers tested three families of models: OPT (Decoder-only), Pythia (Decoder-only), and Flan-T5 (Encoder-Decoder). They varied the size of the models (from 7M to 3B parameters) and the size of the training dataset (from 1,000 to 64,000 fictitious diarists).

Result 1: Scale Drives Self-Knowledge

The first major finding is that the ability to understand knowledge scope is not innate in small models; it emerges with scale.

Three line charts comparing model accuracy across varying parameter counts for OPT, Pythia, and Flan-T5. The graphs show that accuracy generally increases with model size.

In Figure 2, the solid lines represent the standard experimental setup. We can observe distinct behaviors:

  • OPT: Shows a clear trend where increasing model size and dataset size improves performance.
  • Flan-T5: Struggles significantly at smaller sizes but sees a jump in performance once the model reaches a certain scale (around 780M parameters) and is trained on enough data.

Result 2: The “Distributed Information” Penalty

The researchers introduced a control group called the “Simplified Setup” (shown as dashed lines in Figure 2). In this setup, all diary entries for a single person were merged into one long document during training.

The difference was stark. When all information about “Alice” was in a single document, the models had near-perfect accuracy (the dashed lines are at the top). When the information was distributed across different documents (the solid lines), performance dropped.

This suggests that the core difficulty isn’t memorizing the text; it is consolidating information that is dispersed throughout the model’s training data. The “Gap” between these two setups is visualized below:

Bar charts showing the gap in accuracy between standard and simplified setups. The gap is significant for smaller models and datasets but narrows as models scale up.

Figure 3 illustrates that for OPT and Pythia, this gap narrows as the models get larger and are trained on more data. This indicates that larger models are better at “connecting the dots” between separated pieces of memories.

Quantity vs. Quality: Where Do Models Fail?

When a model fails this benchmark, how does it fail? Does it garble the text, or does it simply lose count?

The analysis reveals that models are actually excellent at memorizing the content. If a model decides to recall “Entry #2,” the text of that entry is usually error-free. The failure mode is almost entirely related to counting.

The “Counting” Problem

The researchers plotted the number of documents the model should have recalled vs. the number it actually recalled.

Histograms comparing recalled documents vs. target documents for OPT and Pythia on the 8K dataset. Darker colors (larger models) cluster around the correct number, while lighter colors (smaller models) are scattered.

In Figure 5, look at the diagonal. A perfect model would always land on the diagonal (Target: 3 -> Recalled: 3).

  • Small Models (Lighter colors): They are all over the place. If the target is 3, a small model might recall 1, or it might recall 8. It effectively guesses a random number.
  • Large Models (Darker colors): They cluster tightly around the correct number.

This confirms the paper’s title: Sufficiently large models know how much they know. They stop generating exactly when they run out of valid memories.

The Flan-T5 Anomaly

The encoder-decoder model, Flan-T5, behaved differently. On smaller datasets (8K diarists), it failed to learn the counting mechanism regardless of model size. However, when the dataset size was quadrupled to 32K diarists, the capability suddenly emerged for the larger versions of the model.

Histograms for Flan-T5 showing that on the 8K dataset, recall is inconsistent, but on the 32K dataset, recall improves with scaling.

This suggests that different architectures (Decoder-only vs. Encoder-Decoder) require different “critical masses” of data and parameters to develop this metacognitive trait.

Does Length Matter?

One might assume that recalling 8 documents is harder than recalling 1 because there is more text to generate, increasing the probability of a token error. Surprisingly, the study found that document length and quantity do not impact content accuracy.

Line charts showing Document Accuracy vs. Sentences to Recall. Accuracy remains relatively stable regardless of length for OPT and Flan-T5.

Figure 7 shows “Document Accuracy”—the percentage of recalled documents that are error-free. The lines are remarkably flat. This implies that once a model commits to recalling a document, it can reproduce it perfectly, whether it is short or long. The cognitive bottleneck is not in generating the words, but in deciding which and how many documents to retrieve.

This finding holds true even at the sentence level within a single document. As shown below in Figure 8, larger models (darker colors) correctly recall the exact number of sentences, while smaller models guess randomly.

Histograms showing the number of sentences recalled vs. target sentences. Larger models consistently recall the correct number of sentences.

Why Does This Happen? (Nature vs. Nurture)

The researchers posed a deeper question: Is this capability a result of the model’s architecture, or is it learned during pre-training?

To test this, they took a small Pythia model (which performed poorly) and a small OPT model, and trained them from scratch (random weights) instead of starting with pre-trained weights.

Table comparing pre-trained vs. scratch fine-tuning. Pythia-70M performs significantly better when trained from scratch compared to pre-trained weights.

Table 1 reveals a fascinating contradiction.

  • OPT-125M performed worse when trained from scratch. Its pre-training helped it.
  • Pythia-70M performed better when trained from scratch (jumping from 21% to 45% accuracy).

This suggests that for some models, the pre-trained weights might actually be “resistant” to this specific type of fine-tuning task, potentially due to how they were originally optimized. It highlights that “knowing what you know” isn’t just about raw intelligence; it’s about how the model’s internal representations are structured to handle dispersed information.

Conclusion

This research provides a critical step toward understanding LLM psychology. The authors demonstrated that Large Language Models can indeed understand the scope of their own knowledge, but this is an emergent capability that requires:

  1. Sufficient Scale: Small models simply guess.
  2. Sufficient Data: Models need to see enough examples of “exhaustive recall” to learn the pattern.
  3. Consolidation: Models struggle more when information is fragmented across their training history than when it is contiguous.

The implication for students and practitioners is clear: hallucination often stems from a model’s inability to recognize its own knowledge boundaries. As we build larger models and curate better datasets, we move closer to AI that knows not just how to speak, but when to stop.