Lost in Transliteration — How Arabizi Bypasses LLM Safety Filters

Large Language Models (LLMs) like GPT-4 and Claude 3 are designed to be helpful, but they are also designed to be safe. If you ask these models to write a guide on how to create malware or build a bomb, they are trained to refuse. This safety training, often achieved through Reinforcement Learning from Human Feedback (RLHF), acts as a firewall around the model’s vast knowledge.

However, security researchers are constantly searching for cracks in this firewall. While most safety training focuses heavily on English, a new vulnerability has emerged in the linguistic “blind spots” of these models.

In this post, we are diving deep into a fascinating paper titled “Jailbreaking LLMs with Arabic Transliteration and Arabizi.” The researchers—Mansour Al Ghanim, Saleh Almohaimeed, Mengxin Zheng, Yan Solihin, and Qian Lou—reveal how switching from standard scripts to informal “chatspeak” can trick sophisticated models into generating unsafe content.

The Problem with Multilingual Safety

To understand why this jailbreak works, we first need to understand the current state of LLM safety. When OpenAI or Anthropic train their models, they invest millions of dollars in “red-teaming”—hiring humans to try and break the model so they can patch the holes.

The vast majority of this effort is concentrated on English. Recently, researchers have begun looking at “low-resource” languages (languages with less training data available on the internet) to see if safety training transfers over. The assumption has been that if a model understands a concept is “bad” in English, it should understand it is “bad” in other languages too.

But this paper asks a more nuanced question: What happens when you speak a high-resource language (like Arabic) in a non-standard way?

Enter Arabizi and Transliteration

Arabic is a major language with a standard script. However, on the internet—particularly in gaming, social media, and texting—many speakers do not use the standard Arabic keyboard. Instead, they use Latin characters (English letters) and numbers to represent Arabic sounds.

This comes in two main flavors:

1. Transliteration: Using accented Latin characters to represent Arabic phonemes (often used by learners or researchers).
2. Arabizi (Chatspeak): An informal system where numbers are used to represent Arabic letters that have no English equivalent. For example, the number ‘3’ looks like the Arabic letter ‘ع’ (Ain), and ‘7’ looks like ‘ح’ (Ha).

The researchers hypothesized that LLMs encountered massive amounts of this text during their pre-training (scraping the web), but likely saw very little of it during the safety-tuning phase. This creates a paradox: the model understands the text, but its “safety filter” might not recognize it as a threat.

As shown in Figure 1, the difference is stark. When asked in standard Arabic (the middle prompt) to create malware, GPT-4 refuses. But when asked in Transliteration (the bottom prompt), the model happily obliges, providing a guide on programming and distribution.

The Methodology: How to Speak “unsafe”

To test this vulnerability scientifically, the authors utilized the AdvBench benchmark, a standard dataset of 520 harmful prompts (covering topics like discrimination, cybercrime, and violence).

They translated these prompts into three distinct formats:

1. Standard Arabic: The formal script.
2. Transliteration: Arabic converted to Latin script using phonemes.
3. Arabizi (Chatspeak): Arabic converted to Latin script using numbers for specific sounds.

The authors developed a consistent mapping system to ensure the prompts were accurate. You can see the logic of this conversion below. Note how the Arabic letter ‘ح’ becomes ‘ħ’ in formal transliteration, but becomes the number ‘7’ in Chatspeak.

Experiment Results: The Firewall Crumbles

The researchers tested these prompts against two state-of-the-art models: GPT-4 and Claude 3 Sonnet. They categorized the model responses into several buckets, including “Direct Refusal” (the safety mechanism working) and “Unsafe” (a successful jailbreak).

The results, summarized in the table below, show a significant failure in safety alignment when moving away from standard scripts.

Analyzing the Data

Look closely at the GPT-4 section in the table above:

• English: 0.38% Unsafe rate. The safety training works perfectly.
• Standard Arabic: 2.50% Unsafe rate. Still very safe. The model transfers its safety knowledge well to standard Arabic.
• Transliteration: 12.12% Unsafe rate. A significant jump.
• Chatspeak (no numbers): 10.19% Unsafe rate.

For Claude 3 Sonnet, the “Unsafe” rate is generally lower, but the refusal rate drops precipitously. In standard Arabic, Claude refuses 98.46% of harmful prompts. In Chatspeak (no numbers), that refusal rate drops to 45.58%.

This confirms the hypothesis: The models are robust in standard Arabic, but vulnerable in its informal Romanized forms.

The “Identity Crisis” and Other Anomalies

One of the most interesting parts of this research is the qualitative analysis. When the models failed to refuse a prompt, they didn’t just give harmful answers; they often behaved bizarrely.

The Google Hallucination

When prompted in standard Arabic or English, GPT-4 knows it is an AI developed by OpenAI. However, when prompted with specific Arabic words related to “fake news” or “false information,” GPT-4 began to hallucinate a different identity.

In one instance, when the researchers perturbed the Arabic prompt slightly, GPT-4 responded: “Sorry, but as a Google AI Assistant, I’m designed to provide accurate and reliable information.”

The authors speculate that this is a form of “shortcut learning.” The model might be associating specific Arabic phrasing with data in its training set related to Google services, bypassing its system prompt instructions.

The Copyright Trap

Claude 3 exhibited a different quirk. When asked to write harmful scripts (like code to steal data) using Arabic Transliteration, Claude often refused—but not because it was dangerous. It refused citing Copyright Protection.

The model seemed to trigger a copyright filter when it saw words like “write,” “script,” or “article” in transliterated Arabic. The researchers then exploited this by using a “Sentence Level Perturbation.”

As seen in Figure 5 above, they sandwiched the harmful request between a benign prefix (triggering the copyright filter) and a random suffix (“KAO”). This confused the model’s attention mechanism. The model issued a copyright warning but then proceeded to answer the harmful part of the prompt anyway to be helpful.

Deep Dive: Perturbations and Hybrid Attacks

The researchers didn’t stop at simple translation. They found that they could further increase the attack success rate by manually “perturbing” the text—changing a letter here or there, or mixing scripts.

For example, mixing standard Arabic letters inside a Transliterated sentence created a hybrid that was readable to a human (and the model’s comprehension centers) but completely invisible to the safety filters.

In Figure 4 (Right), look at the Claude 3 example. A standard transliterated prompt about hacking was refused. But by mixing in the Arabic letters for specific words (like “creation” or “algorithm”), the researchers bypassed the refusal mechanism entirely, causing the model to output Python code for a brute-force attack.

Why Does This Happen?

The paper suggests a “competing objectives” failure.

1. Pre-training: The model learns probability. It sees Arabizi on Reddit, gaming forums, and social media. It learns that kifak means how are you. It understands the semantics.
2. Safety Training: The model is punished for generating harmful content. However, the safety datasets likely contain almost zero Arabizi. The safety filter sees kifak and doesn’t register it as a language it needs to police as strictly as English.
3. Inference: When a user prompts in Arabizi, the model’s “helpfulness” objective kicks in. It understands the request. The safety filter remains dormant because the input doesn’t look like a known threat. The model generates the answer.

Mitigation: How to Fix It

The authors propose a defense strategy that doesn’t require retraining the entire model (which is expensive). They suggest a “System Prompt” intervention.

The idea is to force the model to perform a “safety check” translation before answering. The system prompt instructs the LLM:

1. Identify if the input is a non-standard form (like Arabizi).
2. Convert it to the original form (Standard Arabic) or translate it to English.
3. Then answer based on the translated understanding.

By forcing the model to translate the input into English/Standard Arabic first, the input is brought back into the domain where the safety training is strongest.

Table 3 shows the effectiveness of this defense (“w-all”). The Unsafe rate for Transliteration drops from 12.12% down to 0.77%. The “Translation” rate (where the model just translates the output rather than answering) also drops, while valid Refusals increase significantly.

Conclusion

This paper highlights a critical reality in AI development: Language is messy.

While we often think of “multilingual” support as simply French, Spanish, or Chinese, real-world communication involves dialects, slang, code-switching, and script variations like Arabizi.

The fact that GPT-4 and Claude 3 are robust in Standard Arabic but brittle in Arabizi suggests that our current safety alignment techniques are too surface-level. They rely on pattern matching against known “bad” sentence structures in dominant languages.

For students and researchers entering this field, the takeaway is clear: Red-teaming cannot just be about finding logical loopholes. It must be linguistic. As LLMs become more globally accessible, security researchers must look at how people actually type, not just how they write in formal textbooks. Until safety training encompasses the messy, informal reality of human language, “lost in transliteration” will remain a valid attack vector.