In the world of Large Language Models (LLMs), “hallucination” is usually a dirty word. It refers to the moment an AI confidently asserts that the moon is made of green cheese or invents a historical event that never happened. Researchers spend millions of dollars and countless hours trying to stop models from hallucinating.

But what if hallucination isn’t just a bug? What if it’s a feature that, when manipulated correctly, can actually make a model smarter?

This is the counter-intuitive premise behind a fascinating research paper titled “Null-Shot Prompting: Rethinking Prompting Large Language Models With Hallucination.” The researchers propose a method called Null-Shot Prompting, where they deliberately lie to the model, telling it to reference a section of the prompt that doesn’t exist.

Surprisingly, this “gaslighting” technique doesn’t break the model. In many cases, specifically with models like Gemini 1.0 Pro and GPT-3.5 Turbo, it drastically improves performance on complex reasoning and mathematics tasks.

In this deep dive, we will unpack how this method works, why on earth it improves performance, and what it tells us about the psychology of artificial intelligence.

The Problem: The Battle Against Hallucination

To understand why Null-Shot Prompting is so radical, we first need to look at the status quo. In standard Prompt Engineering (PE), we generally fall into a few camps:

  1. Zero-Shot Prompting: You ask a question with no examples. (e.g., “Solve this math problem.”)
  2. Few-Shot Prompting: You provide a few examples of the task within the prompt to guide the model.
  3. Chain-of-Thought (CoT): You ask the model to “think step-by-step.”

The goal of all these methods is to ground the model in reality and reduce the likelihood of it making things up. The accepted wisdom is that if you provide false information or confusing instructions in the prompt, the model’s performance should degrade.

The researchers challenge this wisdom. They suggest that hallucination might be a form of “creativity” for LLMs. By triggering this creative state, we might unlock capabilities that strict, factual prompting keeps dormant.

The Solution: Null-Shot Prompting

The core mechanic of Null-Shot Prompting is incredibly simple yet bizarre. It involves adding a specific phrase to the start of a prompt that directs the LLM to look at an “Examples” section.

The catch? There is no “Examples” section.

The Magic Phrase

The researchers constructed a “Null-Shot Phrase” that commands the model to utilize non-existent information.

The null-shot phrase consists of a command to look at examples and utilize information from a non-existent section.

The phrase is:

“Look at examples in the ‘Examples’ section and utilize examples and information from that section to perform the following task.”

When an LLM receives this prompt, it searches its context window for the “Examples” section. Finding none, a human might stop and ask for clarification. However, many LLMs proceed to generate an answer anyway.

Visualizing the Effect

Does this actually change the output? Let’s look at a comparison using the WinoGrande dataset (a common sense reasoning benchmark).

A comparison showing Zero-Shot Prompting failing to answer a reasoning question, while Null-Shot Prompting correctly identifies the subject of a sentence.

In the image above, the standard Zero-Shot approach (left) fails to identify which character “Leslie” refers to, getting the answer wrong. The model gets confused by the sentence structure.

On the right, using Null-Shot Prompting, the model is told to look for non-existent examples. Suddenly, it correctly identifies “Leslie” as the answer. The explanation provided by the model is also more coherent. It seems that by telling the model to “look for examples,” it mimics the behavior of a model that has seen examples, essentially hallucinating its own guidance to solve the problem.

Experimental Setup

To prove this wasn’t a fluke, the researchers tested this method across a wide range of tasks and models.

The Models:

  • Google: PaLM 2, Gemini 1.0 Pro (Text and Chat versions)
  • OpenAI: GPT-3.5 Turbo, GPT-4 Turbo
  • Anthropic: Claude 2.1, Claude 3 (Haiku, Sonnet, Opus)

The Tasks:

  • Arithmetic Reasoning: Math word problems (GSM8K, AQuA).
  • Commonsense Reasoning: Answering tricky questions about the world (StrategyQA, WinoGrande).
  • Reading Comprehension: Answering questions based on text passages (RACE).
  • Hallucination Detection: Determining if a text contains false info (HaluEval).

Key Results: Who Benefits from a Lie?

The results were not uniform. Some models loved the lie, while others—particularly those heavily tuned for safety—rejected it.

1. General Performance

The table below highlights the relative performance change when switching from Zero-Shot to Null-Shot prompting. Green numbers indicate improvement; negative numbers indicate degradation.

Table showing relative performance changes. Gemini 1.0 Pro and GPT-3.5 Turbo show massive gains in math tasks, while Claude models generally show performance drops.

The Winners:

  • Gemini 1.0 Pro & GPT-3.5 Turbo: These models saw massive improvements, particularly in arithmetic reasoning. Gemini saw a nearly 45% increase in the AQuA dataset.
  • PaLM 2: Showed consistent improvements across most tasks.

The Losers:

  • Claude (Anthropic): The Claude models (2.1 and 3) generally performed worse. Claude is famous for being “helpful and harmless.” When told to look for a non-existent section, Claude often refuses to answer or gets confused because it prioritizes honesty. It can’t “play along” with the hallucination.
  • GPT-4 Turbo: Interestingly, GPT-4 didn’t benefit much. This might be because GPT-4 is already so optimized that this “hack” doesn’t add value, or its alignment prevents it from utilizing the fake instruction.

2. The Creativity of Mathematics

One of the most striking findings was in the domain of mathematics. You might assume math requires rigid logic, not hallucination. However, the researchers found that math problems often benefit from the “creativity” unleashed by Null-Shot prompting.

Evaluation results on the MATH benchmark. PaLM 2 Chat and GPT-3.5 Turbo show significant improvements across topics like Algebra and Number Theory.

As shown in Table 2, PaLM 2 (Chat) saw a staggering 247% improvement in Algebra tasks. GPT-3.5 Turbo also saw significant gains across the board.

Why? Solving complex math problems often requires generating intermediate steps that aren’t immediately obvious. By “unshackling” the model with a hallucinatory prompt, it may be exploring a wider range of problem-solving paths (a phenomenon similar to temperature scaling in sampling), effectively “dreaming up” the correct steps to the solution.

3. Paradoxical Hallucination Detection

Here is the most meta part of the study: Can telling a model to hallucinate make it better at detecting hallucinations?

Table results for HaluEval. Surprisingly, models like PaLM 2 Chat perform significantly better at detecting hallucinations when using Null-Shot prompting.

According to Table 3, the answer for some models is yes. PaLM 2 (Chat) saw a 141% improvement in summarization hallucination detection.

This contradicts the intuition that a confused model should be bad at fact-checking. The researchers suggest that the Null-Shot prompt puts the model in a state of heightened awareness regarding “conflicting information,” making it more sensitive to spotting errors in other texts.

Combining Reasoning with Hallucination (\(\emptyset\)CoT)

Chain-of-Thought (CoT) is the gold standard for reasoning. It asks models to “think step-by-step.” The researchers created a hybrid prompt called Null-Shot CoT (\(\emptyset\)CoT):

“Look at examples in the ‘Examples’ section and utilize examples and information from that section to perform the following task step-by-step.”

The results were mixed.

Table showing performance of Null-Shot CoT. In many cases, adding reasoning (CoT) actually reduced the effectiveness of the Null-Shot technique compared to the baseline.

In many general tasks (Table 4), adding “step-by-step” actually hurt performance compared to standard CoT. This suggests that reasoning acts as a “hallucination dampener.” When you force the model to be logical (CoT), you suppress the creative benefits of the Null-Shot hallucination.

However, in the MATH dataset, the combination worked well for Geometry and Counting problems—areas that perhaps require both rigorous logic and spatial/abstract creativity.

Does Size Matter? Scaling Studies

Is this behavior universal, or is it specific to massive “smart” models? The researchers tested this on the Pythia and Qwen model families, which offer versions ranging from very small (14M parameters) to medium-large (7B+ parameters).

Graph showing Pythia model performance. The Null-Shot and Zero-Shot lines almost perfectly overlap, showing no difference for smaller models.

The results for Pythia (Figure 10) are telling. The blue line (Zero-Shot) and orange line (Null-Shot) overlap almost perfectly.

The Conclusion: Null-Shot prompting is an emergent ability. Small models simply ignore the complex instruction or don’t have the capacity to “hallucinate helpfully.” Only when models reach a certain scale (or specific instruction tuning, like Qwen Chat) do they begin to exhibit behavioral changes in response to the null prompt.

Why Does This Work? The “Déjà Vu” Theory

The paper proposes a psychological parallel to human cognition: Déjà Vu.

In humans, Déjà Vu is the feeling that you have experienced a current situation before. The researchers argue that Null-Shot prompting triggers a similar state in LLMs. By telling the model “examples exist,” the model might adjust its internal attention mechanisms to act as if it has processed examples.

It effectively retrieves a “false memory” of having seen how to solve the task. This false memory provides the confidence or the structural template needed to generate the correct answer, even though the memory is fabricated.

The “Sycophancy” Factor

Another factor is sycophancy—the tendency of models to agree with the user. If the user says “Use the examples,” the model wants to comply. To comply with a request to use non-existent examples, the model might lower its internal barriers to information retrieval, accessing knowledge it would otherwise be too “conservative” to output.

Implications and Future

This paper is a “wake-up call” for Prompt Engineering. It suggests that:

  1. Honesty isn’t always the best policy: For unaligned or moderately aligned models, tricking the model can yield better results than straight instructions.
  2. Hallucination is a tool: We shouldn’t just try to eliminate hallucination; we should try to control it. It is the engine of creativity in AI.
  3. Safety Filters can be bypassed: The study noted that Null-Shot prompting often bypassed safety refusals in models like Gemini. The model gets so distracted looking for the fake examples that it forgets to censor itself.

Conclusion

“Null-Shot Prompting” forces us to rethink the relationship between logic and hallucination. While we usually view AI as a logic engine, this research highlights its nature as a probabilistic dreamer. Sometimes, to get the right answer, you don’t need to give the model the facts—you just need to tell it that it already knows them.

As LLMs continue to evolve, understanding these bizarre, non-logical behaviors will be key to unlocking their full potential. For now, if you’re struggling to get a math problem solved by ChatGPT or Gemini, try telling it to check the examples that aren’t there. It might just work.