Introduction

Imagine having a conversation with Harry Potter about his first Quidditch match, and then, without switching apps or reloading a model, turning to Lord Voldemort to discuss the Dark Arts. While Large Language Models (LLMs) like ChatGPT have mastered open-domain chat, making them truly “stay in character”—especially multiple different characters—remains a significant hurdle.

Current role-playing agents (RPAs) usually face a dilemma. They either rely on prompt engineering (telling the model “Act like X”), which often breaks character over long conversations, or they require training a completely separate model for every single character, which is computationally expensive and inefficient. Furthermore, what happens when you want to add a new character? Usually, you have to retrain the whole system, risking “catastrophic forgetting”—where the model learns the new role but forgets how to play the old ones.

Enter Neeko. Named after the shapeshifting champion from League of Legends, Neeko is a novel framework designed to solve the problem of Multi-Character Role-Playing (MCRP). By leveraging a clever modification of Low-Rank Adapters (LoRA), Neeko can house multiple distinct personalities within a single model, switch between them dynamically, and even learn new characters on the fly without losing previous knowledge.

In this post, we will dissect the research paper behind Neeko, exploring how it uses dynamic architecture to become the ultimate digital method actor.

Background: The Challenge of Digital Acting

To understand why Neeko is necessary, we first need to look at how AI currently handles role-playing.

The Three Traditional Approaches

  1. In-Context Learning (ICL): This is the most common method. You simply prompt the model: “You are Sherlock Holmes.” While easy, the model often slips out of character during long chats because the “persona” is just part of the temporary context, not the model’s weights.
  2. Retrieval Augmented Generation (RAG): The model retrieves character lines or bio information from a database to augment its answers. This helps with factual accuracy (e.g., knowing Sherlock’s address) but often fails to capture the specific tone or speech patterns (like Sherlock’s rapid-fire deduction style).
  3. Fine-Tuning (FT): You train the model specifically on a dataset of the character’s dialogue. This yields the best performance but is rigid. If you want 50 characters, you effectively need 50 models (or one massive model trained on all of them, which confuses the distinct personalities).

The Efficiency Key: LoRA

Neeko builds upon a technique called LoRA (Low-Rank Adapter). Training an entire LLM (billions of parameters) for every character is impossible for most researchers. LoRA freezes the massive pre-trained model weights (\(W_0\)) and injects pairs of small, trainable low-rank matrices (\(B\) and \(A\)) into the network.

Mathematically, the modification to the model’s forward pass looks like this:

Equation for the LoRA forward pass showing how trainable matrices are added to frozen weights.

Here, \(\Delta W = BA\) represents the specific changes needed to adapt the model to a new task (or character). Because \(r\) (rank) is very small, we only train a tiny fraction of the parameters.

However, standard LoRA has a limitation: it’s static. You usually train one LoRA adapter for one task. Neeko asks: What if we could slice up the LoRA adapter and assign different slices to different characters?

The Neeko Framework

Neeko treats the MCRP task as a lifecycle consisting of three stages: Agent Pre-Tuning, Multi-Character Playing (Inference), and Incremental Learning.

The overall framework of Neeko showing Pre-Tuning, Role-Playing, and Incremental Learning phases.

As shown in Figure 1 above, the architecture is designed to keep character data distinct yet accessible within the same model. Let’s break down the technical innovations in each phase.

1. Agent Pre-Tuning: Assigning “Blocks”

In a standard approach, if we wanted a model to learn Harry Potter, Voldemort, and Dobby, we would feed all their dialogues into the model at once. The result? A confused model that might have Voldemort apologizing politely like Dobby.

Neeko solves this using Dynamic LoRA. Instead of one shared matrix for everyone, Neeko divides the low-rank matrices \(B\) and \(A\) into non-overlapping “blocks.”

If we have \(M\) characters, the matrices are sliced such that each character \(k\) owns a specific range of the parameters. The training update for character \(k\) only touches their specific block:

Equation showing how LoRA matrices are sliced into non-overlapping blocks for each character.

By updating only specific indices (\(W_B^k\) and \(W_A^k\)), the gradient descent for one character does not interfere with the parameters of another. This ensures that Harry Potter’s brave tone doesn’t bleed into Voldemort’s malicious speech patterns.

Equation showing the gradient descent update rule applied only to specific character blocks.

2. Inference: The Gating Mechanism

Once the model is trained on multiple characters, how does it know which “block” to use during a conversation?

Neeko employs a Gating Network, inspired by Mixture of Experts (MoE) architectures. During training, the system creates a global role embedding matrix (\(\mathbf{E}_{\text{global}}\)) derived from the profiles of the characters.

When a user interacts with Neeko, the system looks at the input (or the specified role) and calculates a similarity score against the available role embeddings. This generates a set of weights (\(w_k\)) that determine which LoRA blocks to activate.

Equation for the gating mechanism using Softmax to determine role weights.

The Gate ensures that if you are talking to Harry Potter, the “Harry Potter Block” is fully activated. Interestingly, this mechanism also allows for nuance. If a user asks the agent to act like a character that shares traits with two known characters, the gate could theoretically activate a mixture of blocks, though Neeko focuses primarily on precise role selection.

3. Incremental Learning: Handling New Characters

The true test of a role-playing agent is adaptability. What if we want to add Draco Malfoy to the roster after the model is already trained? Re-training the whole system is wasteful. Neeko introduces two strategies to add new roles without “catastrophic forgetting.”

Strategy A: Fusion (For Limited Data)

If we don’t have enough dialogue data to train a new character from scratch, Neeko constructs the new character by fusing existing ones.

It calculates how similar the new character is to the existing ones (using the embedding matrix) and creates a weighted combination of the existing LoRA blocks. For example, a new villain might be constructed as 60% Voldemort’s block and 40% of another antagonist’s block.

Equation showing the Fusion strategy where a new weight matrix is a weighted sum of existing character blocks.

Strategy B: Expansion (For Abundant Data)

If there is sufficient data for the new character, Neeko simply expands the matrix. It adds a new block to the LoRA adapter. To prevent destroying previous knowledge, Neeko freezes the old blocks and the old gating parameters, training only the new block and the expanded dimensions of the gate.

Equation showing the minimization objective for the Expansion strategy, optimizing only new parameters.

This theoretically allows Neeko to learn an unlimited number of characters over time, bounded only by memory.

Experiments and Results

To validate Neeko, the researchers used the Character-LLM-Data dataset and compared Neeko against standard baselines including LLaMA-2 (using ICL and RAG) and standard LoRA fine-tuning.

They proposed a comprehensive set of evaluation metrics categorized into three groups:

  1. Character Consistency: Does it behave and speak like the character?
  2. Knowledge Consistency: Does it know the character’s world (Virtual Knowledge) without hallucinating, while maintaining Real World knowledge?
  3. Dialogue Consistency: Is the conversation stable and relevant?

Neeko vs. The Baselines

The results, summarized in Table 1 below, show that Fine-Tuning (FT) methods generally beat prompt-based methods (ICL/RAG). However, Neeko stands out even among fine-tuning methods.

Table 1: Comparison results of distinct role-playing agents. Neeko generally scores highest across average metrics.

Neeko achieves the highest stability scores. This confirms that separating character parameters into distinct blocks prevents the “personality bleed” often seen in standard LoRA models where all characters share the same parameters.

The Incremental Challenge

The researchers then tested the “unseen character” scenario. They trained the agents on 8 characters and then introduced a 9th one.

Figure 2: Radar charts comparing Neeko against ICL, RAG, and standard LoRA. Neeko covers the largest area.

As seen in the radar charts above (Figure 2), Neeko (the orange line) consistently envelops the baselines.

  • Neeko vs. ICL/RAG: Prompting methods struggle with Behavior and Utterance style because they haven’t “learned” the character deep down.
  • Neeko vs. Standard LoRA: Standard LoRA suffers when trying to learn a new character incrementally because updating the weights for the new guy messes up the weights for the old crew. Neeko’s independent blocks solve this.

Human Evaluation

Automated metrics are useful, but human judgment is the gold standard for conversation. The researchers recruited human evaluators to score the responses.

Violin plots showing the distribution of human evaluation scores. Neeko (red) shows consistently higher density at the top of the scale.

The violin plots in Figure 3 visually demonstrate the dominance of Neeko (in red). The bulk of the distribution is shifted towards the higher scores (6-7) compared to ICL (blue), RAG (orange), and LoRA (green), particularly in avoiding hallucinations and maintaining virtual knowledge consistency.

Resource Efficiency

One might assume that maintaining separate blocks for characters makes Neeko bloated. However, the study reveals that Neeko remains highly efficient.

Table 5 comparison of training time and memory usage. Neeko is comparable to LoRA and much cheaper than Character-LLM.

As shown in Table 5, Neeko’s memory usage (13.55 GB) is almost identical to standard LoRA (13.49 GB) and vastly superior to Character-LLM (107.84 GB), which requires full fine-tuning. This makes Neeko a viable solution for consumer-grade hardware, whereas other methods require massive server clusters.

Conclusion and Implications

Neeko represents a significant step forward in the field of personalized AI. By formalizing the task of Multi-Character Role-Playing (MCRP) and solving it with Dynamic LoRA, the authors have created a framework that is both versatile and efficient.

The key takeaways from this research are:

  1. Modularity works: Breaking down neural network adapters into character-specific blocks prevents interference and preserves distinct personalities.
  2. Adaptability is key: The ability to fuse existing knowledge or expand architectures allows AI to grow over time rather than remaining static.
  3. Efficiency matters: Neeko achieves these results without a significant increase in computational cost compared to standard parameter-efficient fine-tuning.

As we move toward a future where AI agents serve as companions, tutors, and gaming NPCs, architectures like Neeko will be essential in ensuring these digital entities act with the consistency, depth, and variety that users expect. The days of the “one-personality-fits-all” chatbot may soon be behind us.