Ouroboros: Breaking the Speed Limit of LLMs with Phrase-Based Speculative Decoding

Large Language Models (LLMs) have revolutionized how we interact with information, but they suffer from a persistent bottleneck: latency. If you have ever watched ChatGPT type out an answer word by word, you have experienced the limitations of autoregressive decoding. Because every new token depends on the previous one, models must generate output sequentially. This process is slow and computationally inefficient, leaving expensive GPUs idle while waiting for memory access.

To solve this, researchers developed Speculative Decoding, a technique that uses a smaller model to “draft” text and a larger model to verify it. However, even Speculative Decoding hits a ceiling. If the draft is too short, you don’t save much time. If it’s too long, the drafts often fail verification, wasting effort.

Enter Ouroboros, a novel framework presented by researchers from Tsinghua University and ModelBest Inc. Ouroboros reimagines the drafting process by moving from tokens to phrases. By recycling discarded verification results and parallelizing the drafting phase, Ouroboros achieves speedups of up to 3.9x over standard decoding—without requiring any additional model training.

In this post, we will deconstruct how Ouroboros works, the mathematics behind its efficiency, and why it represents a significant leap forward in LLM inference.

The Bottleneck: Why Drafting is Hard

Before diving into Ouroboros, we must understand the “Goldilocks problem” of Speculative Decoding (SD).

In a standard SD setup, you have two models:

1. Draft Model (\(S\)): Small and fast, but less accurate (e.g., Llama-7B).
2. Target Model (\(T\)): Large and accurate, but slow (e.g., Llama-70B).

The Draft Model predicts the next few tokens (say, 5 tokens). The Target Model then processes all 5 tokens in a single parallel forward pass to verify them. If the draft is correct, you just generated 5 tokens in the time it usually takes to generate one.

However, efficiency depends heavily on the draft length (\(\gamma\)) and the draft model size.

As shown in Figure 1, there is a difficult trade-off:

• Left Chart: Larger draft models are more accurate (acceptance rate goes up) but are slower to run.
• Right Chart: Longer drafts offer higher potential speedup, but they take longer to generate and are more likely to contain errors that cause the verification to fail.

The authors identified two critical limitations in current SD methods:

1. Insufficient Drafting: Generating drafts token-by-token is still too slow.
2. Underutilized Computation: When a draft fails (e.g., the target model rejects the last 3 tokens), that work is usually discarded entirely, even if it contained semantically useful phrases.

The Ouroboros Solution

Ouroboros addresses these issues by shifting the atomic unit of drafting from a single token to a phrase. It introduces a recursive mechanism where the model learns phrases from its own verification inputs (hence the name “Ouroboros,” the mythical snake eating its own tail).

The framework introduces four key innovations:

1. Accelerating Drafting: Generating candidate phrases in parallel.
2. Lengthening Drafts: Stitching phrases together to create longer candidates at near-zero cost.
3. Recycling Verification: Turning discarded tokens into useful phrases for the future.
4. Context Reuse: Using phrases from prompt history.

Let’s look at the high-level comparison between standard Speculative Decoding and Ouroboros:

In Figure 2, the top section shows standard SD generating tokens one by one (“I”, “like”, “to”…). The bottom section shows Ouroboros using pre-computed phrases (“I like to do”, “waste my time on”). Instead of running the draft model forward six times for six tokens, Ouroboros might only need two forward passes to construct the same sentence by chaining these phrases.

The Mathematics of Speedup

To understand the theoretical gain, let’s look at the speedup equation for standard Speculative Decoding:

Here, \(A(\gamma)\) is the average accepted tokens, \(t_T\) is the target model time, and \(t_S\) is the draft model time.

Ouroboros improves this equation in two ways. First, it reduces the cost of drafting by a factor of \(c\) (via phrase acceleration). Second, it extends the draft length by \(\beta\) (via phrase concatenation) without extra computation. The new speedup equation becomes:

The goal of Ouroboros is to maximize \(c\) and \(\beta\) while keeping overhead low.

Core Method: Phrase-Based Drafting and Verification

The heart of Ouroboros lies in how it constructs and verifies these longer drafts.

1. Lengthening Drafts via Phrases

In modern GPUs, LLM generation is memory-bound, not compute-bound. This means verifying 10 tokens takes roughly the same time as verifying 1 token, because the bottleneck is moving model weights into memory, not the math itself.

Ouroboros exploits this. Instead of submitting just one draft to the target model, it constructs multiple longer drafts by appending different candidate phrases to the current generation.

Let’s say the draft model generates a sequence ending in \(d_\gamma\). Ouroboros looks up \(K\) phrases that start with \(d_\gamma\) and creates \(K\) different candidate sequences:

This looks like a lot of work for the target model, but the authors use a clever Attention Masking trick to verify all these candidates in a single forward pass.

As visualized in Figure 3, the attention mask allows the Target Model to verify the “Prefix” and “Draft” normally. Then, it creates parallel branches for “Phrase 1” and “Phrase 2.” The model computes the probability of the tokens in Phrase 1 independently of Phrase 2. This is physically one batch on the GPU, incurring almost no extra time cost compared to verifying a single draft.

2. Generating Phrases from Verification

Where do these phrases come from? Surprisingly, many come from the “failures” of previous iterations.

In standard Speculative Decoding, if the target model rejects the suffix of a draft, those tokens are deleted. However, the authors observed that even rejected tokens often contain correct sequences—they were just misplaced or slightly offset.

Using a matching function (Equation 10), Ouroboros identifies segments in the discarded draft that actually match the target model’s output distribution.

Figure 4 illustrates this perfectly.

• Left (Standard SD): The draft outputs [1, 2, 4, 6, 8]. The target expects [2, 4, 6, 8]. Because the draft started with 1 (incorrect), standard SD rejects everything after it. The model has to start over, wasting the correct sequence 4, 6, 8.
• Right (Ouroboros): The system detects that 6, 8 was a valid phrase, just in the wrong spot. It saves 6, 8 as a phrase. In the next step, it applies this phrase immediately, skipping the need to generate those tokens one by one again.

Experiments and Results

The researchers evaluated Ouroboros on standard benchmarks including HumanEval (Code), GSM8K (Math), and CNN/DM (Summarization). They compared it against Vanilla decoding, Lookahead decoding, and standard Speculative Decoding.

Speed Comparison

The results are stark. Ouroboros consistently outperforms all baselines.

In Figure 5, look at the blue bars (Ouroboros). On the HumanEval dataset with the Yi-34B model, Ouroboros reaches 61.2 tokens/second, compared to just 15.6 for vanilla decoding and 21.5 for standard speculative decoding. This is a massive leap in throughput.

This performance holds up across different task types, as seen in Table 2:

For the Llama-2-70B model on GSM8K (math reasoning), Ouroboros achieves a 2.68x speedup over vanilla decoding, significantly higher than the 1.88x achieved by standard speculative decoding.

Block Efficiency

A key metric introduced in the paper is Block Efficiency (\(\eta\)), which measures the theoretical upper bound of acceleration by calculating the ratio of generated tokens to target model calls.

Table 13 shows that Ouroboros achieves much higher block efficiency (generating more tokens per call). For HumanEval, it achieves an efficiency of 13.12, compared to 11.16 for Speculative Decoding and just 3.08 for Lookahead. This confirms that the phrase-based approach is successfully getting the target model to accept larger chunks of text at once.

Ablation Studies

Does every part of the system help? The authors broke down the contributions of each component:

As shown in Table 4:

• Baseline speed: 21.46 token/s
• Adding Phrase Drafting: 49.90 token/s (The biggest jump)
• Adding Phrase Lengthening: 55.92 token/s
• Adding Verification Recycling: 58.18 token/s
• Adding History Reuse: 61.20 token/s

While phrase drafting provides the massive initial boost, the subsequent optimizations (lengthening and recycling) squeeze out significant additional performance.

Conclusion

Ouroboros demonstrates that we haven’t yet hit the physical limits of Large Language Model inference speed. By moving away from the rigid token-by-token paradigm and embracing phrase-based generation, we can better utilize the massive parallel computing power of modern GPUs.

The most compelling aspect of Ouroboros is that it is training-free. It does not require distilling a new draft model or fine-tuning the target model. It is an algorithmic improvement that can be applied to existing drafting-verification pipelines immediately.

As LLMs continue to grow in size and capability, techniques like Ouroboros that optimize the “software” side of inference—making the decoding process itself smarter—will be essential for making these models deployable and responsive in real-world applications.