If you have ever played with a “base” language model—one fresh out of pre-training—you know it can be a bit of a wild card. It might ramble, complete your sentence instead of answering your question, or output something unsafe. To turn these raw computational engines into helpful assistants like ChatGPT or Llama-Instruct, we need alignment.

Alignment is the process of fine-tuning a model to follow instructions and adhere to human preferences. While we have general ideas about how this works—usually a mix of Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF)—the specific “secret sauce” used by big labs (like OpenAI or Meta) is often kept closed-source.

This lack of transparency makes it hard for students and researchers to know what actually matters. Is it the dataset size? The specific algorithm? The hyperparameters?

In this post, we are breaking down a paper from Columbia University titled “LIONS: An Empirically Optimized Approach to Align Language Models.” The researchers conducted a rigorous audit of the modern alignment pipeline. They didn’t just propose a new complex algorithm; they ran controlled experiments to find the best configuration for every step of the training process.

By the end of this article, you will understand the three-stage pipeline that creates state-of-the-art models and see how the researchers used these findings to create “LION” models that outperform official industry baselines using only open-source data.

The Three-Stage Pipeline

Before we get into the optimizations, let’s establish the standard framework. The researchers focus on a modern three-stage pipeline widely used in the industry:

  1. Supervised Fine-Tuning (SFT): Teaching the model to answer questions using a dataset of instruction-response pairs.
  2. Offline Preference Learning: Using a dataset of “better vs. worse” responses to steer the model toward what humans like, usually via Direct Preference Optimization (DPO).
  3. Online Preference Learning: Generating new responses with the model, judging them, and updating the model iteratively.

The researchers analyzed each stage in isolation to find the optimal settings. Let’s walk through them.

Stage 1: Supervised Fine-Tuning (SFT)

The first step is teaching the model the format of a conversation. The objective is simple: maximize the probability of the correct response given a prompt.

Equation for standard log likelihood maximization.

However, the implementation details matter immensely. The researchers investigated three technical choices:

  • Padding: Extending short sequences with “blank” tokens so all data fits a fixed length.
  • Packing: Stitching multiple short examples together into one long sequence to fill the context window efficiently (common in pre-training).
  • Loss Masking: Calculating the error (loss) only on the model’s response, ignoring the user’s prompt tokens. (You don’t want the model to learn how to write the prompt; you want it to learn how to answer).

The Findings

The researchers tested these strategies on Gemma-2b. The results were clear:

Performance comparison for different SFT strategies on OpenLLM and Arena-Hard-Auto*.

Key Takeaway: The combination of Packing + Loss Masking is superior.

Looking at the table above, you can see that simply adding “Loss Masking” boosts the Arena-Hard (chat capability) score significantly. But when you combine it with Packing on a large dataset (1.6M examples), the score jumps to 8.8, nearly double the baseline.

Why does this happen? The researchers hypothesize that standard padding might cause the model to overfit to the “shape” of chat templates. Packing forces the model to attend to the actual content more robustly, while loss masking ensures the model focuses its learning capacity strictly on generating answers.

Stage 2: Offline Preference Learning (DPO)

Once the model knows how to chat (via SFT), we need to teach it what creates a good response. This is typically done using Direct Preference Optimization (DPO).

DPO uses a dataset of triplets: a prompt (\(x\)), a winning response (\(y_w\)), and a losing response (\(y_l\)). The goal is to increase the likelihood of the winner and decrease the loser.

The DPO Loss Function.

This stage is where the paper offers its most extensive insights. They tested four critical variables: sequence length, reference models, the “beta” hyperparameter, and dataset scaling.

1. Sequence Length Matters

In the standard implementation of DPO (like in the Hugging Face alignment handbook), a sequence length of 1024 is often used. The researchers found that simply doubling this to 2048 yields a significant performance boost. Complex reasoning and chat history take up space; cutting them off hampers alignment.

2. Choosing the Reference Model

DPO requires a “reference model” (usually the SFT model) to ensure the new model doesn’t drift too far away from the original language distribution. The researchers tested using stronger models or updated models as the reference but found that the standard SFT model works perfectly fine as a reference. Complexity isn’t always better here.

Table showing the effect of training with longer sequences and different reference models.

3. Tuning Beta (\(\beta\))

The \(\beta\) parameter controls the strength of the KL-divergence constraint—essentially, how much the model is allowed to deviate from the reference model.

A common question in DPO training is: “Does the optimal Beta change if I use more data?”

Chart showing Beta vs Performance across dataset sizes.

The finding: The optimal \(\beta\) (around 0.1) stays consistent regardless of dataset size. This is great news for practitioners because you can tune your hyperparameters on a small subset of data (saving compute) and confidently apply them to your massive full training run.

4. The Scaling Laws of DPO

Perhaps the most interesting finding in this paper is the relationship between data size and training steps. In pre-training, we know that “more data = better.” The researchers found the same holds true for alignment.

Graph showing saturation points of DPO based on dataset size.

Take a look at Figure 1 above.

  • Small Data (Blue Line): Performance peaks early and then degrades (overfitting/over-optimization).
  • Large Data (Green Line): Performance continues to climb for much longer and reaches a higher peak.

This suggests that DPO follows scaling laws. You cannot simply train longer on a small dataset to get better results; you will just overfit. To get a better model, you need a larger preference dataset.

5. Quality vs. Quantity (Data Filtering)

There is a trend in recent research suggesting “Less is More”—that a small, highly curated dataset is better than a large, noisy one. The researchers put this to the test by comparing random sampling against sophisticated filtering algorithms (like DEITA, Alpagasus, and Argilla).

Comparison of data filtering methods vs simple random sampling.

The Verdict: While filtering helps when you are restricted to small data budgets (10k samples), simply using more data (100k random samples) beats every fancy filtering method.

This challenges the “small but high-quality” narrative. In offline preference learning, diversity and quantity seem to trump strict quality filtering.

Optimization Tip: Efficient DPO

To make training on these large datasets feasible, the researchers introduced a technical optimization. Standard DPO requires padding tokens to batch data, which wastes compute. They implemented a version that removes padding and concatenates sequences, using FlashAttention to handle the boundaries.

Illustration of efficient DPO implementation without padding.

This trick alone improved training speed by nearly 27%.

Stage 3: Online Preference Learning

The final stage is Online DPO. In Offline DPO, we use a static dataset. In Online DPO, the model acts as a chatbot, generates new responses to prompts, and an external judge (like a Reward Model) ranks them. The model then learns from its own generated data.

Is this computationally expensive step worth it?

Table showing the effect of Online DPO.

The results in Table 5 show that Online DPO (ODPO) significantly improves performance on chat benchmarks (Arena Hard Auto), though it doesn’t change much for core knowledge tasks (OpenLLM).

This makes sense: Online DPO helps the model refine its style and instruction-following capabilities using its own outputs, but it doesn’t teach the model new facts (since it’s generating the data itself).

Putting It All Together: The LION Series

Armed with these empirical insights, the researchers created a “recipe” for the LION models:

  1. SFT: Packing + Loss Masking.
  2. Offline DPO: Large dataset (mixing UltraFeedback, HelpSteer, etc.), Sequence Length 2048, tuned Beta.
  3. Online DPO: One iteration of generating and ranking responses.

They applied this recipe to two base models: Gemma-2b and LLaMA-3-8b.

The Results

The comparison is striking. They compared their LION models against the “official” instruct versions released by Google (Gemma-it) and Meta (LLaMA-3-it), which were trained with massive internal resources and proprietary data.

Table comparing LION models against official baselines.

Key Achievements:

  • Gemma-2b-LION outperforms the official Gemma-2b-it and even beats larger models like LLaMA-2-7b-chat on key benchmarks.
  • LLaMA-3-8b-LION-ODPO achieves an Arena-Hard win rate of 22.0, beating the official LLaMA-3-8b-instruct (20.6).

This proves that an optimized pipeline using only open-source data can rival or exceed the proprietary pipelines of major tech companies.

What Does a “Well-Aligned” Model Look Like?

To visualize why their models worked better, the authors analyzed the “probability margin.” This is the difference in confidence the model has between the winning response and the losing response (\(\pi_\theta(y_w) - \pi_\theta(y_l)\)).

Violin plots showing probability margins.

  • Undertrained (Left): The model can barely distinguish winners from losers; the distribution is centered near zero.
  • Overtrained (Right): The model is extremely confident, pushing margins to the edges, likely overfitting.
  • Best Performing (Middle): It forms a parabolic shape. The model improves confidence on pairs it already knew (right side of the parabola) but also learns to distinguish pairs it previously got wrong (left side). This balance is the signature of a healthy alignment process.

Conclusion and Implications

The “LIONS” paper is a breath of fresh air for the open-source community. It moves away from the “black box” alchemy of LLM training and provides a clear, evidence-based recipe.

The Recipe for Success:

  1. Don’t ignore SFT details: Sequence packing and loss masking are non-negotiable.
  2. Scale your Preference Data: For DPO, quantity and diversity (100k+ samples) often beat heavy filtering.
  3. Go Online: If you want top-tier chat performance, you must perform online preference learning.
  4. Optimize: Simple hyperparameter tuning (like \(\beta\)) translates well across scales.

By following these steps, the researchers demonstrated that the gap between open-source and closed-source alignment is smaller than we thought—in fact, with the right optimization, open-source might just be winning.

For students and practitioners, this paper serves as a vital handbook: before inventing a new complex algorithm, make sure you’ve optimized the basics.