Introduction

In the current landscape of Artificial Intelligence, Large Language Models (LLMs) like GPT-4 and Llama 2 are the undisputed kings. They write code, compose poetry, and answer complex queries. But underneath the hood, these models share a common constraint: Tokenization.

Before an LLM sees your text, a “tokenizer” chops sentences into discrete numbers (tokens). While efficient, this process strips away the visual richness of language. It struggles with complex PDFs, non-standard layouts, and “visually rich” text like emojis or mixed scripts. Furthermore, tokenization creates a “vocabulary bottleneck”—if a word or character isn’t in the model’s pre-defined dictionary, the model struggles to process it.

But what if a model didn’t need a dictionary? What if it could just “look” at the text, pixel by pixel, the way a human reads a scanned document?

In this deep dive, we explore a fascinating research paper titled “Autoregressive Pre-Training on Pixels and Texts”. The researchers introduce PixelGPT and DualGPT, models that abandon traditional tokenization in favor of processing raw images of text. They demonstrate that teaching an AI to predict the next “patch” of pixels can result in powerful language understanding, potentially solving the multilingual struggles of current LLMs.

Background: The Tokenization Trap

To understand why PixelGPT is revolutionary, we first need to understand the status quo. Traditional LLMs process text as a sequence of discrete IDs.

1. Input: “Hello 🐱”
2. Tokenizer: [15496, 243, 102] (Hypothetical IDs)
3. Model: Processes these numbers.

This works well for English, but it breaks down in several scenarios:

• Multilingualism: A tokenizer optimized for English might chop a Thai or Chinese sentence into inefficient, nonsensical fragments, making it harder for the model to learn patterns.
• Visual Context: If you bold a word, highlight it in red, or use a specific font to convey sarcasm, a standard tokenizer throws that information away. It only sees the letters.
• OCR Dependency: To read a PDF or a screenshot, we usually run Optical Character Recognition (OCR) first. If the OCR makes a mistake (reading “rn” as “m”), the LLM consumes the error.

Enter Pixel-Based Modeling

Previous researchers attempted to solve this with models like PIXEL. These models treated text as images but used an “encoder-only” architecture (similar to BERT) and processed images in grayscale. They were good at understanding but couldn’t generate text fluently in a sequence.

The paper we are analyzing today takes the next massive leap: Autoregressive Modeling on Pixels. This means building a model that reads visuals left-to-right and predicts what comes next, unlocking the generative capabilities that made GPT famous.

The Core Method: Teaching Transformers to See

The researchers propose two primary architectures: PixelGPT (trained only on images) and DualGPT (trained on both images and text). Let’s break down how they work.

1. Rendering Text as Rich Images

Unlike previous approaches that used 8-bit grayscale or binary (black/white) images, PixelGPT renders text as 24-bit RGB images.

Why does this matter? Because modern communication is colorful. We use emojis, syntax highlighting in code, and colored text for emphasis. By using RGB, the model can “see” that a red “WARNING” is different from a black “WARNING.”

The text is rendered onto a long strip, which is then chopped into small squares called patches.

As shown in Figure 8 above, the sentence is rendered (even with emojis!) and sliced. A black patch is added at the end to signal the “End of Sequence” (EOS).

2. The Architecture: PixelGPT

PixelGPT uses a Transformer Decoder, the same architecture used by Llama 2. However, the input isn’t a list of word tokens; it’s a sequence of image patches.

Here is the process illustrated in Figure 1:

1. Render: The text “My cool cat…” is turned into an image.
2. Patchify: The image is cut into 16x16 pixel patches.
3. Linear Projection: Each patch is flattened and projected into a vector embedding.
4. Transformer Layers: The model processes these embeddings.
5. Next Patch Prediction: This is the critical change. The model tries to predict the pixels of the next patch in the sequence based on the previous ones.

The mathematical objective is to minimize the difference between the predicted pixels and the actual pixels. Specifically, they calculate the probability of the sequence as the product of conditional probabilities:

In this equation, \(x_p\) represents the visual patches. The model tries to maximize the likelihood of patch \(t\) given all previous patches (\(1\) to \(t-1\)).

3. DualGPT: The Best of Both Worlds

While PixelGPT is impressive, text is still a very dense, efficient way to store information. The researchers hypothesized that a model could benefit from “bilingual” training—learning from both raw pixels and discrete text tokens.

Figure 2 shows the architecture of DualGPT. It has a shared Transformer Decoder backbone but uses two different “heads” (output layers):

• Classification Head: For text inputs, it predicts the next token ID (standard GPT behavior).
• Regression Head: For image inputs, it predicts the pixel values of the next patch.

This allows the model to transfer knowledge between modalities. It can learn the high-level semantic logic from text and the fine-grained visual details from images.

Experiments & Results

The researchers put these models through a gauntlet of tests, primarily using the GLUE benchmark (for English understanding) and XNLI (for cross-lingual understanding).

1. Can a Pixel Model Rival a Text Model?

The first question is simple: Can PixelGPT, which has no concept of “words” or “letters,” actually understand language?

Table 2 provides the answer: Yes.

• Beating GPT-2: PixelGPT (317M parameters) achieves a GLUE average score of 74.2, which is comparable to (and on several tasks better than) GPT-2.
• Beating PIXEL: It outperforms the previous state-of-the-art pixel model (PIXEL) on difficult tasks like RTE (Recognizing Textual Entailment) and WNLI.

This proves that tokenization is not strictly necessary for high-level language understanding. A neural network can learn to “read” directly from visual input.

2. The Multilingual Superpower

The most exciting result comes from multilingual testing. Because PixelGPT doesn’t use a specific language tokenizer, it doesn’t suffer when switching between English, Chinese, Arabic, or Thai. It just sees different shapes.

Figure 10 (the radar chart) illustrates this dominance.

• Look at the spike for Thai (THA) and Chinese (ZHO). PixelGPT (the green line) significantly outperforms BERT (pink) and PIXEL (orange).
• In Thai, PixelGPT scored +11.3 points higher than BERT.

This confirms the “Vocabulary Bottleneck” hypothesis: standard models fail on languages with complex scripts or tokenization rules, but PixelGPT glides right through them because it treats all languages as visual patterns.

3. Scaling Laws: The Data Hunger

One of the most important findings for the future of this technology is how it scales. Does feeding it more data make it smarter?

Figure 3 shows the performance curves as training data increases (x-axis):

• TextGPT (Purple): Starts strong but flattens out.
• PixelGPT (Blue): Starts much lower (it’s harder to learn pixels than tokens). However, notice the steep upward slope. It crosses the PIXEL baseline and keeps climbing.

This suggests that pixel-based models are data-hungry. They need more compute to get started, but their performance ceiling might be higher because they aren’t constrained by a fixed vocabulary.

4. Seeing the Invisible: Emojis and Color

Finally, the researchers tested if the RGB rendering actually helped. They used the HatemojiBuild dataset, which detects hate speech that relies on emoji context.

Figure 7 shows why RGB is vital.

• Case 1: “can we all agree that 💀 is 🌿”. In grayscale, the model misses the context. In RGB, it identifies the sequence correctly.
• Case 2: “Muslims are so full of 😡”. The red face is a strong emotional signal.

The RGB-trained PixelGPT outperformed grayscale versions by +2.7 accuracy points on this dataset, proving that color provides semantic signal, not just decoration.

Conclusion and Implications

The paper “Autoregressive Pre-Training on Pixels and Texts” challenges a fundamental assumption of NLP: that text must be processed as abstract numbers.

By treating text as images, PixelGPT demonstrates that we can build models that are:

1. Robust to “noisy” text: They can read weird fonts, colors, and layouts.
2. Truly Multilingual: They don’t need language-specific tokenizers, performing exceptionally well on non-Latin scripts like Thai and Chinese.
3. Scalable: While they require more training data initially, they show a strong scaling trend that rivals text-based models.

DualGPT takes this a step further, showing that combining the efficiency of text tokens with the robustness of pixels yields the best results, effectively smoothing out the “modality competition.”

What’s Next?

The authors note that generation is still a hurdle. Currently, the model predicts patches of pixels. To get text back out, you essentially have to run OCR on the generated image patches. Future work will likely focus on making the output stage more seamless, perhaps by having the model generate text tokens based on visual inputs directly.

This research paves the way for a future where AI reads the internet exactly as we do: visually, in full color, and without a dictionary.