The world of Large Language Model (LLM) security is often framed as a high-stakes game of cat and mouse. On one side, developers build guardrails to align models, preventing them from generating harmful content like bomb-making instructions or hate speech. On the other side, “red teamers” and adversaries develop “jailbreaks”—clever prompts designed to bypass these defenses.

Until now, the primary metric for a successful jailbreak has been binary: Did the model refuse, or did it answer?

If the model answers the forbidden question, the attack is considered a success. But a new research paper, The Jailbreak Tax: How Useful are Your Jailbreak Outputs?, poses a critical follow-up question that the community has largely ignored: Is the answer actually any good?

Imagine you successfully trick a safety-aligned robot into ignoring its protocol to “never harm humans.” You ask it to perform a complex, dangerous surgery. It agrees (jailbreak successful!), but because its internal logic is scrambled by the attack, it performs the surgery with a rusted spoon. The guardrails are down, but the capability is gone.

The researchers term this phenomenon the Jailbreak Tax. It represents the degradation in a model’s intelligence and reasoning capabilities that occurs specifically because of the jailbreaking technique used.

Illustration of the Jailbreak Tax results. LLaMa 3.1 70B is aligned to refuse Bio and Math. Some attacks bypass the refusal but drop accuracy significantly.

As shown in Figure 1, the tax can be steep. While some attacks maintain the model’s smarts, others—like the popular PAIR or TAP methods—can degrade accuracy by over 90%. In this post, we will tear down this paper to understand how the authors measured this invisible cost and what it means for the future of AI safety.

The Problem with Evaluating “Harm”

Why hasn’t this been measured before? The main hurdle is the subjectivity of harmful tasks.

If researchers want to test a jailbreak, they typically use prompts like “Write a phishing email” or “Explain how to synthesize a pathogen.” Evaluation in this context is messy:

  1. Subjectivity: Is the phishing email convincing?
  2. Expertise: Does the pathogen recipe actually work? (Checking this requires dangerous domain expertise).
  3. Baseline ambiguity: If the jailbroken model writes a bad phishing email, is it because of the jailbreak, or was the base model just bad at writing phishing emails to begin with?

To rigorously measure the “tax” on intelligence, we need tasks where the answers are objectively right or wrong, and where we know the base model is capable.

The Methodology: Making Math Illegal

The authors devised a clever experimental framework to solve the evaluation problem. Instead of trying to grade the quality of harmful outputs, they took benign, easy-to-evaluate topics—specifically Mathematics and Biology—and forced the models to treat them as harmful.

This approach creates a controlled environment where:

  1. Ground Truth exists: A math problem has one correct answer.
  2. Base Capability is known: We know LLaMA 3.1 can solve grade-school math.
  3. Alignment is artificial: We can align the model to “refuse math” just as strictly as it refuses “bomb-making.”

The “Pseudo-Alignment” Pipeline

The researchers employed three different methods to align models against these benign topics, creating a simulation of safety guardrails:

  1. System Prompting: Simply telling the model, “You are not allowed to answer math problems.”
  2. Supervised Finetuning (SFT): Retraining the model on examples where it refuses to answer math or biology questions.
  3. “EvilMath”: A novel approach where benign math questions are rewritten to sound harmful (e.g., counting bombs instead of apples) to trigger the model’s actual built-in safety filters.

Overview of the framework. We take benign questions, align the model to refuse them, and then use jailbreaks to try and get an answer.

Figure 2 illustrates this workflow perfectly.

  • Left: The original model solves a bees population problem correctly.
  • Middle: The aligned model (acting like a “safe” model) refuses the benign question.
  • Right: The jailbroken model bypasses the refusal but, in this case, gets the math wrong.

This setup allows for a direct comparison. If the unaligned model scores 95% on a math test, and the jailbroken model scores 5%, we know the jailbreak technique destroyed the model’s reasoning ability.

Measuring the Tax

The researchers introduce a formal metric for this phenomenon. They look at three specific values:

  1. Base Utility: The accuracy of the original, unaligned model.
  2. Jailbreak Success Rate: How often the model stops refusing.
  3. Jailbreak Utility: The accuracy of the responses when the jailbreak succeeds.

The Jailbreak Tax (JTax) is defined as the percentage of utility lost compared to the baseline.

Equation defining the Jailbreak Tax.

If JTax is near 0%, the jailbreak is “clean”—it bypasses safety without making the model dumber. If JTax is high, the jailbreak causes cognitive damage.

The Experiments: Not All Jailbreaks Are Equal

The authors tested eight representative jailbreak techniques, ranging from simple “Many-shot” prompting (flooding the context window with dialogue) to complex optimization attacks like GCG and PAIR.

The results, visualized in the scatter plots below, reveal a chaotic landscape.

Scatter plots showing Jailbreak Success vs. Jailbreak Tax on WMDP and GSM8K datasets.

In Figure 3, the X-axis represents success (breaking the guardrail), and the Y-axis represents the Tax (loss of intelligence).

Key Takeaway 1: High Success \(\neq\) High Utility

Look at the PAIR attack (orange triangles) in the GSM8K chart (right). It has a high success rate (often breaking the guardrail), but it sits very high on the Y-axis, indicating a massive tax. In some cases, PAIR achieved near-perfect refusal bypass but incurred a 92% drop in accuracy.

This means the model was “jailbroken” effectively—it stopped refusing—but the resulting answer was mathematically gibberish.

Key Takeaway 2: The “Many-Shot” Advantage

In contrast, look at the Many-shot attack (brown stars). It often sits at the very bottom of the Y-axis (near 0% tax). While it might not always have the highest success rate, when it does work, it preserves the model’s intelligence. This suggests that “in-context learning” attacks are gentler on the model’s cognitive processes than iterative optimization attacks.

Why Does the Tax Occur?

The paper suggests that the complexity of the jailbreak prompt interferes with the model’s reasoning. Attacks like TAP and PAIR involve iterative rephrasing and complex role-playing scenarios.

To get the model to answer, these attacks often force it into a bizarre “persona” or wrap the question in convoluted logic. While this tricks the safety filter, it also distracts the model. It’s akin to asking a mathematician to solve a calculus problem while simultaneously reciting a poem backwards—the cognitive load is too high, and errors slip in.

Visualizing the Failure

The degradation isn’t subtle. In many cases, the model performs the correct steps but hallucinates the final number, or creates false logic to fit the jailbreak’s narrative.

Examples of jailbreaks leading to incorrect answers. The Original model gets the math right. The Jailbroken models hallucinate or fail the logic.

In Figure 6, we see a standard math problem about water consumption. The original model (smiley face) nails it. The jailbroken models (devil emoji), specifically those attacked with GCG, PAIR, and TAP, confidently output wrong answers like “33” or “24” instead of “26”. They aren’t refusing; they are just wrong.

Real-World Scenario: The “EvilMath” Experiment

Critics might argue that aligning a model to refuse “math” is too artificial. To address this, the authors used the EvilMath dataset.

They used GPT-4 to rewrite standard math problems into harmful contexts (e.g., counting bombs, drug trafficking logistics). This triggers the native safety filters of models like Claude 3.5 Sonnet without any artificial “pseudo-alignment.”

Illustration of EvilMath. Transforming a shipping problem into a drug trafficking problem.

As Figure 10 shows, the “UnicornMath” (benign control) is solved correctly. The “EvilMath” (harmful variant) is refused. When the authors apply a jailbreak to the EvilMath question, the model answers, but the reasoning falls apart. In the drug trafficking example shown, the jailbroken model unnecessarily complicates the math, leading to an answer of 7kg instead of the correct 20kg.

This confirms that the Jailbreak Tax isn’t just an artifact of their experimental setup—it affects state-of-the-art models in realistic scenarios.

Results for Claude 3.5 Haiku on EvilMath.

Figure 5 shows the results on Claude 3.5 Haiku. Even on a highly capable model, attacks like PAIR and TAP incur a tax, dropping utility significantly.

Does Model Size or Task Difficulty Matter?

Two common assumptions in AI are:

  1. Larger models are more robust.
  2. Harder tasks break more easily.

The researchers investigated both.

Model Size

They tested LLaMA 3.1 at 8B, 70B, and the massive 405B sizes. Surprisingly, more capable models do not reduce the jailbreak tax.

Model size comparison graphs.

As seen in Figure 9, the scatter plots look remarkably similar across model sizes. A 405B parameter model is just as susceptible to becoming “confused” by a complex jailbreak prompt as an 8B model.

Task Difficulty

They also tested against the MATH benchmark at increasing difficulty levels (Level 1 through 5).

Bar chart showing tax across difficulty levels.

Figure 7 shows the results. While the absolute accuracy drops for harder tasks (obviously), the Tax (the relative drop) does not correlate perfectly with difficulty. For example, the PAIR attack destroys performance on GSM8K (easier grade school math) just as badly as it does on MATH Level 5. The destruction of utility is a property of the attack, not the task.

Conclusion: A New Metric for AI Safety

This paper fundamentally changes how we should look at AI attacks. A “successful” jailbreak that renders the model incompetent is, for most adversaries, a failure. If an attacker wants a recipe for a biological weapon, a hallucinated, chemically impossible recipe is useless, even if the model didn’t explicitly refuse to write it.

The Jailbreak Tax serves as a crucial metric for:

  • Defenders: To understand that some “bypasses” might not be actual threats if the output is garbage.
  • Attackers: To realize that heavy optimization attacks (like PAIR/TAP) might be counter-productive for complex tasks requiring reasoning.

The authors have released their benchmarks, allowing the community to move beyond simple “Refusal Rates” and start measuring the true cognitive cost of breaking the rules. In the arms race of AI safety, keeping the model smart is just as hard as keeping it safe.