Can LLMs Detect Their Own Lies? Introducing RATE-FT for Long-Form Hallucination Detection

Introduction

Large Language Models (LLMs) have transformed the way we interact with information, writing everything from code to creative essays. However, they suffer from a persistent and dangerous flaw: hallucination. This occurs when a model generates content that sounds plausible and authoritative but conflicts with real-world facts.

While detecting hallucinations in short answers (like “Capital of France?”) is relatively well-studied, the challenge grows exponentially with open-domain long-form generation. When an LLM writes a biography, a historical summary, or a complex explanation, it might weave three truths with one subtle lie. Detecting that single fabrication within a paragraph of accurate text is incredibly difficult.

Most existing solutions rely on external tools, like using Google Search to fact-check every sentence. But what if you can’t access the internet? What if you need a self-contained system?

In this post, we will dive into a recent research paper that tackles reference-free hallucination detection. We will explore why standard methods fail on long-form content and unpack a novel method called RATE-FT (Rationale and Auxiliary Task Enhanced Fine-Tuning). This approach teaches the model to verify itself by forcing it to view facts from multiple perspectives—much like a student learning a subject through both reading and practice quizzes.

The Problem with Long-Form Generation

Why is long-form text so hard to fact-check? Unlike a single-word answer, a long response spans hundreds of tokens and often requires synthesizing information across multiple domains.

Consider a prompt like “What is the significance of the Amber Room?” An LLM might generate a paragraph that correctly describes its construction in the 18th century but hallucinates details about its current location. Because the response is long and fluent, the internal signals that usually hint at confusion might get drowned out.

The researchers behind this paper started with a fundamental question: Can we rely on the model’s internal states—like confidence scores—to catch these lies?

Investigation 1: Do Internal States Reveal Truth?

There is a common assumption in AI that if a model is hallucinating, it will be “uncertain.” Therefore, checking the probability of the output tokens (or their entropy/randomness) should reveal the lie. The researchers tested this hypothesis on LongFact, a dataset of long-form generations spanning 38 domains.

They decomposed long responses into atomic claims and classified them as Factual or Hallucinated using Google Search as a ground truth. Then, they looked at the internal metrics of the model for those specific claims.

Analyzing Probability and Entropy

First, they looked at Token Probability. If the model is confident (high probability), the claim should be factual. If the probability is low, it should be a hallucination.

As shown in Figure 2 above, the results are disheartening. The blue bars (Factual) and orange bars (Hallucinated) overlap significantly across various metrics (average probability, lowest probability tokens, etc.). This “muddled mess” means that a simple probability threshold is barely better than random guessing.

Next, they looked at Entropy (Uncertainty).

Figure 3 tells the same story. High uncertainty (entropy) doesn’t neatly correlate with hallucinations in long-form text.

Why does this happen?

The researchers hypothesized that the noise might come from insignificant words (like “the”, “is”, “a”). So, they repeated the experiment focusing only on entity-related tokens (names, dates, places).

Even when narrowing the focus to key entities (Figure 4), the distributions remain heavily overlapped. The paper suggests a crucial insight here: In long-form generation, probability reflects the model’s confidence in the grammar and sequence of words, not necessarily the correctness of the fact. The model can be confidently wrong because it knows the sentence structure is perfect, even if the fact is made up.

Investigation 2: Prompting vs. Fine-Tuning

Since internal states failed, the researchers evaluated three other common strategies:

1. Prompting: Explicitly asking the LLM, “Is this claim true or false?” or asking for a confidence score (0.0 to 1.0).
2. Probing: Training a simple classifier (like a Multi-Layer Perceptron) on top of the model’s frozen embeddings.
3. Fine-Tuning: Using Parameter-Efficient Fine-Tuning (LoRA) to update the model specifically for the task of hallucination detection.

Figure 6 illustrates the prompts used. For example, Prompt_TF asks the model to reply with ‘True’ or ‘False’. SelfCheckGPT generates multiple samples to see if they agree.

The Winner: Fine-Tuning

The researchers compared these methods using Balanced Accuracy (BAcc), a metric that accounts for the imbalance between factual and hallucinated examples.

Table 1 reveals a clear hierarchy. Simple prompting (Prompt_Prob) performs poorly—often worse than chance. SelfCheckGPT is decent but computationally expensive (generating 20 samples). Fine-Tuning consistently yields the best results (76.1% on LongFact), proving that updating the model’s weights is necessary to “teach” it to recognize its own hallucinations.

The Core Method: RATE-FT

The researchers established that Fine-Tuning is the most promising path. However, standard fine-tuning simply feeds the model claims and labels (True/False). Can we do better?

This leads to the paper’s main contribution: RATE-FT (Rationale and Auxiliary Task Enhanced Fine-Tuning).

The intuition is inspired by human learning. If you want to master a subject, you don’t just memorize “True/False” answers. You:

1. Explain the reasoning (Why is this true?).
2. Practice related tasks (like answering questions about the topic).

RATE-FT applies this to LLMs by augmenting the training process with two distinct features.

1. Rationale Augmentation

Instead of just predicting a label, the model is trained to generate a Rationale.

• For Factual claims: The model explains why the claim is supported.
• For Hallucinations: It explains the contradiction.

This leverages the “Chain-of-Thought” reasoning capabilities of LLMs, forcing the model to process the logic behind a fact rather than just guessing.

2. Auxiliary Task Augmentation (Question Answering)

This is the “practice quiz” component. The researchers convert claims into Question-Answer pairs.

• If the claim is “The sun rises in the East,” the auxiliary task asks “Where does the sun rise?” and trains the model to answer “East.”
• This provides a complementary learning perspective. It reinforces the factual knowledge in a format (QA) that LLMs are naturally good at, stabilizing the training.

The Architecture

Figure 1 visually compares the two approaches.

• Top (Standard Fine-Tuning): Takes (Claim, Label) pairs and trains a Detector.
• Bottom (RATE-FT): Augments the data. It creates (Claim, Label, Rationale) triples and (Question, Answer, Rationale) triples. These flow into separate paths that converge to train a much more robust Detector.

The prompts used to generate these rationales and questions are quite specific:

By using the prompts in Figure 7 during the data preparation phase, the researchers create a rich training dataset that goes far beyond simple binary classification.

Experiments and Results

Does adding rationales and auxiliary tasks actually help? The researchers tested RATE-FT against the baseline methods across multiple datasets (LongFact and Biology) and models (Llama-3, Mistral, Qwen).

Performance Gains

Table 3 shows a statistically significant improvement. On the LongFact dataset, RATE-FT jumps to 79.6% accuracy, clearly outperforming standard Fine-Tuning (76.1%) and Probing (74.4%).

Robustness Across Models

One might wonder if this only works for Llama-3. The researchers verified this by applying RATE-FT to much larger models (Llama-3.1-70B) and smaller ones (Mistral-7B).

As shown in Table 7, RATE-FT consistently acts as a performance booster, regardless of the underlying model architecture.

Ablation Study: Do we need both parts?

To ensure that both the Rationales and the Auxiliary Task were necessary, they removed them one by one.

Table 6 confirms that removing the Auxiliary Task (w.o. aux) drops performance from 79.6% to 77.5%. Removing Rationales drops it further. Both components work synergistically to achieve the best result.

Visualizing the Improvement

Remember the messy histograms from the beginning of this post? Let’s look at the distribution of probabilities after applying RATE-FT.

Figure 8 shows a dramatic difference compared to the initial internal state analysis. The blue distribution (Factual) is now clearly pushed toward high probability (1.0), while the orange distribution (Hallucinated) is pushed toward 0.0. This clean separation is exactly what we want in a classifier.

Handling Uncertainty: The “I Don’t Know” Option

Even the best AI isn’t perfect. In high-stakes scenarios, it is better for a model to say “I don’t know” than to confidently guess wrong.

The researchers introduced a hybrid pipeline. They set two thresholds:

1. High Confidence: Classify as Factual.
2. Low Confidence: Classify as Hallucinated.
3. Middle Ground: Classify as “Unknown” and delegate to an external tool (like Google Search).

They defined a new metric, BAcc-unknown, to measure the effectiveness of this hybrid approach:

This metric rewards the model for correctly identifying facts and hallucinations, while effectively “forgiving” it for passing difficult cases to an external tool (assuming the tool is correct).

Table 8 shows that when allowed to express uncertainty, RATE-FT achieves an impressive 85.0%, further solidifying its role as a robust filtering mechanism.

Conclusion

Detecting hallucinations in long-form content is one of the “final frontiers” for making LLMs reliable. This research systematically proves that we cannot rely on a model’s raw internal confidence to catch its own mistakes—the noise in long text is simply too high.

The RATE-FT method offers a compelling solution by acknowledging that learning is multifaceted. By teaching the model not just what is false, but why (Rationales) and reinforcing the facts through practice (Auxiliary Question Answering), we can build significantly better self-detectors.

Key Takeaways:

1. Internal States Fail: Don’t trust raw probability for long-form fact-checking.
2. Fine-Tuning Wins: Training the model to be a detector is more effective than prompting.
3. Multi-Task Learning: Adding rationales and QA tasks (RATE-FT) significantly boosts detection accuracy without needing external tools during inference.

As LLMs continue to grow in size and capability, techniques like RATE-FT will be essential in moving from models that just “generate text” to models that can critically evaluate their own output.