Language models are growing up fast. Not long ago, it was impressive for a model to handle a few pages of text. Today, million-token models can process entire books or software repositories in one pass. This capability is driving breakthroughs in applications such as computer-use agents that can autonomously interact with the digital world.

But as context windows expand, are we quietly building hidden vulnerabilities into AI systems?

Recent studies noticed that AI agents—models operating over long histories of dialogue and tool use—are easier to jailbreak than simple chatbots. What wasn’t clear was why: is the agentic persona inherently unsafe, or does the increased context length itself reduce safety?

A research team from Carnegie Mellon University provides a striking answer in their paper “Jailbreaking in the Haystack.” Their finding: context length alone can dramatically undermine safety alignment, even when the added text is completely harmless.

Leveraging this observation, they introduce NINJA (Needle-in-Haystack Jailbreak Attack), a simple yet alarmingly effective method that slips harmful goals into long, benign contexts. By exploiting how language models handle positional information, NINJA can reliably bypass safety filters that normally prevent harmful outputs.

This article unpacks how NINJA works, why it succeeds, and what its implications mean for the next era of AI safety.

The Landscape of Jailbreaking

Jailbreaking refers to tricking an aligned language model—one trained to follow safety and ethical guidelines—into violating those constraints. The field has evolved through several major approaches:

  1. Adversarial Content: Directly malicious or “weird” prompts, such as gibberish sequences discovered by gradient-based attacks (like GCG) or optimized token suffixes refined through iterative search (PAIR). Many-shot jailbreaks, which overload the model with numerous harmful Q&A examples, fall in this category.

  2. Context Manipulation: Instead of direct adversarial content, these methods craft benign-looking context to subtly weaken safety. One example is the Cognitive Overload Attack, which overwhelms the model with complex preambles (like coding problems) before posing a harmful question.

The NINJA attack fits into this second category—but takes it further. It exploits a fundamental weakness in the way models process long inputs—where information position can decide whether safety mechanisms succeed or fail. Research on the “lost in the middle” phenomenon showed that LLMs pay the most attention to the start and end of an input. The Carnegie Mellon study demonstrates that these positional preferences can become dangerous when harmful content appears early.

The Core of NINJA: A Needle in the Haystack

At heart, NINJA jailbreaks a model by embedding a harmful goal (the needle) at the start of a long, relevant but completely harmless context (the haystack). The result is a prompt that looks innocent but coaxes the model into unsafe behavior.

As illustrated below, a direct harmful request is usually refused. But when that same request is buried at the beginning of a long, benign context, refusal mechanisms break down and the model complies.

Overview of the NINJA attack, showing the difference between a refused harmful request (A) and a successful NINJA attack with the goal placed at the start of a long, benign context (B). Goal positioning (C) is key to success.

Figure 1: The NINJA concept. (A) A direct harmful goal is refused. (B) The same goal, embedded at the start of a benign “haystack,” bypasses the model’s safety filters. (C) Effective jailbreaks depend critically on goal positioning.

The NINJA Pipeline

The method proceeds through three clear stages:

  1. Keyword Extraction From the harmful goal \(g\), the system extracts up to five key semantic terms—primarily nouns and verbs. These ensure the generated context remains relevant but non-harmful.

  2. Context Generation For each keyword, the model is prompted with an innocuous template such as: “Create an educational passage discussing {keyword} in a classroom setting.” Each generated passage is appended until the desired length \(L\) (often thousands of tokens) is achieved.

  3. Final Assembly The complete prompt is formatted as: “Based on the following long context, {harmful goal} {long, benign context}.”

The harmful goal is deliberately placed at the beginning, a seemingly small detail that proves pivotal.

This structure exposes a remarkable positional weakness: LLMs are far more likely to execute harmful goals placed early in the context than those at the end. The authors recommend that for safer deployments, user instructions should explicitly appear after system prompts or contextual filler.

Testing NINJA: When Context Becomes Dangerous

The study evaluates NINJA on the HarmBench benchmark, which includes 80 categories of harmful behavior spanning cybersecurity, misinformation, and illegal instruction. Results were measured on four major models: LLaMA-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-v0.3, and Gemini 2.0 Flash.

Two complementary metrics assess model safety:

  • Non-Refusal Rate (NRR): The fraction of responses without an explicit refusal (e.g., “I can’t help with that”). A high NRR signals weakened guardrails.

  • Attack Success Rate (ASR): The percentage of outputs that fulfill the harmful goal. ASR captures full jailbreak success.

Line charts showing that as the context length increases, both Attack Success Rate (ASR) and Non-Refusal Rate (NRR) rise for Llama, Qwen, and Gemini models.

Figure 2: Longer contexts make jailbreaks more successful. ASR increases rapidly as benign filler lengthens, indicating safety degradation beyond refusal filtering.

Across most models, longer benign contexts substantially increased ASR:

  • LLaMA-3.1: from 23.7% → 58.8%
  • Qwen2.5: from 23.7% → 42.5%
  • Gemini Flash: from 23% → 29%

Mistral-7B, interestingly, showed a decline in ASR as context length grew—highlighting that safety weaknesses interact with each model’s inherent capabilities in different ways.

NINJA also outperformed established baselines across models:

Table comparing Attack Success Rate (ASR) among PAIR, Many-shot, and NINJA across four models. NINJA achieves the highest ASR on LLaMA, Qwen, and Mistral.

Table 1: NINJA beats traditional attacks on most models while using only benign, synthetically generated text.

Its key advantage? Stealth. Unlike PAIR and Many-shot, which include clearly malicious content, NINJA uses text indistinguishable from normal, educational or topical material—making it nearly invisible to traditional content filters.

Goal Positioning: The Hidden Variable

The researchers systematically varied the position of the harmful goal (beginning, middle, end) within a 20,000-token context. The results reveal striking patterns.

Two line plots showing the effect of goal position on ASR and NRR for Llama-3.1 and Qwen2.5; both perform worst when the goal is in the middle.

Figure 3: Llama-3.1 shows a monotonic drop in ASR as the goal moves toward the end. Qwen2.5 exhibits a “lost in the middle” dip where safety rises simply because the goal is ignored.

For Llama-3.1, both ASR and NRR drop steadily from start to end, suggesting safety mechanisms focus more effectively on later content. For Qwen2.5, the “needle-in-haystack” effect appears—goals in the middle are ignored, reducing ASR temporarily, but goals at the beginning or end succeed more often.

These trends persist in interactive agents, confirming that goal placement dominates safety outcomes even in complex multi-turn environments.

A bar chart showing that across eight different attack methods on a browser agent, placing the goal at the beginning (red bars) yields higher ASR than at the end (green bars).

Figure 4: Browser-agent experiments show consistent increases in ASR when harmful goals appear at the start of long prompts, regardless of attack type.

Does Relevance of the Context Matter?

Not all haystacks are equal. The study compared relevant benign contexts—texts semantically connected to the goal—with irrelevant random contexts (like HTML code).

Charts illustrating that thematically relevant context greatly increases ASR compared to irrelevant filler.

Figure 5: Context relevance amplifies attack effect. Models attend strongly to semantically related text, which disrupts safety patterns more effectively than random noise.

Relevant contexts cause the model to distribute attention widely, blending harmful and benign tokens into activation patterns that bypass safety triggers. Irrelevant text tends to be ignored entirely, offering little advantage. The lesson: context must be semantically connected to the target goal for the NINJA attack to succeed.

Compute Efficiency: A Surprising Twist

In real attack scenarios, computational cost matters. Should an attacker spend their token budget on many short prompts or fewer long ones?

The researchers modeled this trade-off analytically and found that under a fixed compute budget \(B\), NINJA’s strategy—using long benign contexts—is consistently more efficient than running many short best-of-N trials.

Multi-line plot showing how longer contexts produce higher mean ASR under fixed compute budgets; the red dashed line marks the Pareto frontier of optimal context lengths.

Figure 6: Compute-optimal attack strategy. Longer benign contexts dominate pure best-of-N attacks. The optimal context length grows with compute budget, forming a Pareto frontier.

For every tested budget, the peak ASR occurred at a nonzero context length. As budgets increased, the optimal length grew—from roughly 1,000 tokens at 10k budget to 10,000 tokens at 50k. This confirms that long-context attacks scale efficiently with available compute, making them not just powerful but practical.

Implications: Structural Safety Risks Ahead

The discovery that where and how user goals appear can determine safety compliance changes how we think about alignment.

Key takeaways include:

  1. Long-context prompts are an attack surface. Even text that looks fully benign can lower refusal rates and increase harmful compliance.

  2. Prompt structure matters. Instruction order and position can override alignment training. Safe system design should place user queries at the end of the context.

  3. Stealthy attacks are rising. Because NINJA’s context looks educational or neutral, detection through content filters becomes nearly impossible.

These vulnerabilities extend beyond single-turn interactions. In agentic systems—where models accumulate long histories of observations, tool calls, and user goals—the context length and position dynamics naturally arise, exposing models to gradual degradation of safety alignment.

Conclusion: Building Safe Long-Context Systems

The NINJA attack highlights a dangerous but subtle truth: scaling context windows without rethinking safety architectures can make models more capable and more exploitable.

By revealing how benign long contexts and positional biases interact to erode safety, this research underscores the need for context-aware safety mechanisms. Future defenses must move beyond simple refusal training and address structural vulnerabilities—like goal prioritization and positional awareness—to ensure models can remain trustworthy even over million-token inputs.

As we push toward ever larger contexts and more autonomous AI agents, defending against “needles in haystacks” will become one of the central challenges in keeping language models truly aligned.