Imagine teaching a robot to make coffee. It learns the steps, gets good at it, and you are happy. Now you teach it how to toast bread. After a few lessons it has mastered toasting — but when you ask it to make coffee again, it has forgotten. This kind of catastrophic forgetting — where learning a new skill erases an old one — is a stubborn problem for modern AI.

Deep Reinforcement Learning (DRL) has scored spectacular wins in recent years: mastering Go and StarCraft, controlling complex physical systems, and helping design molecules. But most DRL systems are trained for one task in a static environment. When the environment or goal changes, they often must start from scratch. Continual Reinforcement Learning (CRL) is the research frontier aimed at fixing this: enabling agents that learn over a lifetime, retain past skills, and leverage them to learn faster in the future.

This article is a guided tour of a recent comprehensive survey on CRL. We’ll unpack the main ideas, the challenges, how researchers classify methods, common benchmarks, and the most promising directions ahead. The goal is to give you a clear mental model of the field so you can quickly understand research papers, pick appropriate baselines, or design your own experiments.

Figure 1 gives the high-level setting: an agent encounters tasks one after another and must perform well on both new and previously seen tasks.

A diagram illustrating the continual reinforcement learning setup where an agent learns a sequence of tasks and is tested on all previously seen tasks after each training phase.

Figure 1: The general setting for Continual Reinforcement Learning (CRL). An agent learns tasks sequentially and is evaluated on its ability to perform all tasks seen so far.

What follows is structured so you can pick your entry point:

  • Quick refresher on the building blocks (RL and CL)
  • What makes CRL different from related paradigms
  • The main challenges: a triangular trade-off
  • How we evaluate lifelong learners
  • A practical taxonomy: what kind of knowledge is stored/transferred
  • Walkthrough of major method classes with representative ideas
  • Benchmarks, scenarios, and applications
  • Open questions and future directions

If you’re a practitioner who wants actionable takeaways, focus on the taxonomy and the benchmark sections. If you’re a student, the background and metrics will ground your understanding.

A short (but precise) primer

Before diving into continual variants, let’s recap the essentials in a way that directly supports later sections.

Reinforcement learning (RL) problems are commonly modeled as Markov Decision Processes (MDPs). A policy π maps states to actions; the fundamental objective is to maximize expected discounted return. In notation:

\[ V_\pi = \mathbb{E}_{\tau \sim P_\pi}\left[\sum_{t=0}^{H-1} \gamma^t R(s_t, a_t)\right], \]

where τ is a trajectory sampled under π, γ is the discount factor, and H is episode length. Algorithms come in roughly two families:

  • Value-based: learn value functions such as Q(s,a) (e.g., Q-learning, DQN).
  • Policy-based / actor-critic: directly optimize the policy parameters (e.g., PPO, SAC).

Continual Learning (CL), on the other hand, focuses on learning from a sequence of tasks without catastrophic forgetting. It emphasizes the stability–plasticity trade-off: remain stable enough to preserve old knowledge yet plastic enough to learn new skills. The three canonical families of CL techniques are:

  • Replay-based: store or generate past data and replay it during learning.
  • Regularization-based: penalize parameter updates that would break prior knowledge.
  • Parameter-isolation (modular/mask-based): reserve separate parameters per task or selectively activate network parts.

CRL brings these CL ideas into RL’s interactive and often non-stationary setting.

How CRL fits with other multi-task approaches

CRL sits in a spectrum of multi-task paradigms. Figure 2 visualizes the differences.

A comparison of four reinforcement learning paradigms: Traditional RL (one agent, one task), Multi-Task RL (one agent learns multiple tasks simultaneously), Transfer RL (sequential transfer), and Continual RL (one agent learns sequentially over time).

Figure 2: A visual comparison of RL paradigms. CRL is unique in its focus on a single agent learning over a long, sequential timeline.

Key distinctions:

  • Multi-Task RL (MTRL): learn a fixed set of known tasks simultaneously. Task identities are available and the training is typically joint.
  • Transfer RL (TRL): focus on accelerating a target task using source-task experience; forgetting source tasks is not the main concern.
  • Continual RL (CRL): tasks arrive sequentially; the agent must continually adapt while preserving prior competence. This is strictly more general: it includes both transfer concerns and retention.

CRL is especially important for embodied agents and robotics, where the world keeps changing and repeated retraining is infeasible.

The central challenge: a triangular balance

In supervised CL we usually talk about the stability–plasticity dilemma. CRL adds a third, critical axis: scalability. Designing a continual RL system means balancing:

  • Stability — preserve performance on past tasks (avoid forgetting).
  • Plasticity — learn new tasks efficiently and enable forward transfer.
  • Scalability — scale to many tasks under limited memory and compute.

These three interact: aggressive parameter isolation preserves stability but hurts scalability; heavy replay buffers help stability but can be impractical on long task streams; overly strong regularization preserves stability but reduces plasticity. The trade-offs are summarized below.

A triangular diagram illustrating the balance between Plasticity, Stability, and Scalability in CRL.

Figure 3: The triangular balance of challenges in CRL. A successful CRL agent must manage the trade-offs between these three aspects.

Designing methods that hit an acceptable balance for your application (robotics, game environments, dialogue agents) is the core engineering decision for CRL.

Measuring lifelong learning: metrics that matter

Standard episodic reward alone is insufficient to evaluate continual behavior. CRL borrows and adapts CL metrics. Consider N tasks and p_{i,j} be the normalized performance on task j after training through task i.

Important metrics:

  • Average performance (A_i):

    \[ A_i := \frac{1}{i}\sum_{j=1}^i p_{i,j}, \]

    and A_N summarizes final average performance across all N tasks.

  • Forgetting (FG_i):

    \[ FG_i := \max(p_{i,i} - p_{N,i}, 0), \]

    which measures how much performance on task i has dropped after later training. Average FG over tasks gives an overall forgetting score.

  • Forward transfer (FT): whether earlier tasks help learning later ones. One definition:

    \[ FT_i := \frac{1}{N-i}\sum_{j=i+1}^N \left(p_{i,j} - p_{i-1,j}\right). \]

  • Backward transfer (BT): whether learning later tasks helps prior tasks.

Beyond task-centric metrics, practitioners should measure efficiency: model size (final parameter count), sample efficiency (number of environment steps needed to reach target performance), and computational cost (training time). When applying CRL to real systems, these resource metrics are often the ultimate constraint.

Tasks, benchmarks, and realistic scenarios

CRL research uses a variety of task suites, each probing different aspects:

  • Navigation / grid-worlds (MiniGrid): simple, easy to scale and useful for rapid prototyping.
  • Continuous control / robotics (MuJoCo, Meta-World, Continual World): require precision and handle continuous state-action spaces.
  • Video games (Atari, Procgen, StarCraft): high-dimensional visual input and complex long-horizon planning.
  • Real robotics and embodied control: sensor noise, hardware constraints, and safety concerns show up here.

Benchmarks differ in observability, number of tasks, task length, and whether task identities are available. Table 1 (in the original survey) compares several popular CRL benchmarks — pick the one whose constraints match your application. There is no single “best” benchmark yet.

Important scenario types (Table 2 in the survey):

  • Lifelong adaptation: evaluated mainly on future tasks; emphasizes adaptability.
  • Non-stationarity learning: reward or dynamics change over time; agent evaluated on all tasks.
  • Task incremental learning: task ids are known and used (easier setting).
  • Task-agnostic learning: the hardest setting — no task ids, boundaries are unknown.

For real-world deployment, task-agnostic scenarios are most realistic but also the most demanding.

A table comparing various continual reinforcement learning benchmarks across features like environment type, number of tasks, and evaluation metrics.

Figure 4: A comparison of modern CRL benchmarks. Each benchmark tests different aspects of continual learning.

A table formalizing different CRL scenarios based on their learning and evaluation processes.

Figure 5: Formal comparison of common CRL scenarios (task-aware vs task-agnostic, non-stationarity forms, and evaluation protocols).

A practical taxonomy: what knowledge is stored and transferred?

The survey’s central organizing idea is simple and useful: classify CRL methods by the type of knowledge they store or transfer. In RL, core knowledge types are:

  • Policy (and value functions)
  • Experience (trajectories, transitions)
  • Dynamics (models of the environment)
  • Reward (goal descriptions, shaping functions)

Each leads to a family of CRL methods. The taxonomy helps you map a new algorithm to a concrete design pattern and to identify complementary combinations.

A diagram illustrating the general structure of a CRL method, showing the four knowledge types: policy, experience, dynamics, reward.

Figure 6: The general structure of CRL methods, centered on four knowledge types an agent can store and transfer.

Below we walk through each family, highlight canonical ideas, and point out typical strengths and trade-offs.

1) Policy-focused methods

Policy-focused approaches concern what to memorize or adapt in the agent’s policy/value networks. They break down into three subtypes:

  • Policy reuse: store whole policies (or policy library) and reuse them to initialize new tasks, improve exploration, or compose to form new behaviors. Advantages: fast jumpstarts when tasks are similar. Drawbacks: storing full policies scales poorly.

A practical reuse strategy is optimistic or max-based initialization of Q-values:

\[ \hat{Q}_{\max}(s,a) = \max_{M \in \hat{\mathcal{M}}} Q_M(s,a), \]

where \(\hat{\mathcal{M}}\) is a set of observed tasks; this encourages optimistic exploration.

  • Policy decomposition: represent policies as shared components plus task-specific factors. Techniques include latent bases (PG-ELLA style), multi-head architectures, modular networks, and hierarchical skill libraries. This improves scalability while enabling transfer. An example decomposition is: \[ \theta_k = L s_k, \] where L is a shared latent basis and s_k are task-specific coefficients.

Modular and hierarchical decompositions (e.g., Progressive Neural Networks, modular composition, skill libraries) are particularly compelling for complex embodied tasks.

  • Policy merging: compress knowledge from multiple task-specific policies into a single network using distillation, masks, hypernetworks, or regularization (e.g., EWC). Distillation trains a “student” network to match outputs of previous policies; hypernetworks generate task-conditional weights; masks selectively activate parameters per task.

Elastic Weight Consolidation (EWC) is a frequently used regularization baseline:

\[ \mathcal{L}_{\mathrm{EWC}} = \mathcal{L}_{\text{task}} + \sum_i \frac{\lambda}{2} F_i (\theta_i - \theta_i^\star)^2, \]

where the Fisher information F_i estimates parameter importance after previous tasks.

Policy-focused methods are the most common in the literature. Choose reuse or decomposition when task similarity is high and you want positive forward transfer; choose merging and regularization when memory is limited and task identities are noisy.

The framework of policy reuse: store a policy library and use it to initialize, explore, or compose new policies.

Figure 7: Policy reuse — old policies are saved and reused to initialize or compose new policies.

The framework of policy decomposition: shared latent basis, multi-head, modular, and hierarchical decomposition strategies.

Figure 8: Policy decomposition — separating shared components from task-specific ones to promote transfer and reduce interference.

Policy merging techniques: distillation, masks, hypernetworks used to compress multiple policies into a single model.

Figure 9: Policy merging — consolidate knowledge into compact representations using distillation, masks, or hypernetworks.

2) Experience-focused methods

Experience-focused approaches store (or generate) experience tuples to rehearse past behaviors. They split into:

  • Direct replay: keep a replay buffer with selected transitions from earlier tasks, then interleave them with new-task experience. CLEAR and selective replay strategies fall here. Strengths: simple and effective; they provide concrete examples to prevent forgetting. Weaknesses: memory and privacy concerns.

  • Generative replay: instead of raw data, train a generative model (VAE, GAN) to sample pseudo-experiences for older tasks. This saves memory but depends on the generative model’s fidelity. In high-dimensional visual domains, maintaining realistic generative models is challenging but worthwhile when privacy or storage is constrained.

Experience methods are often the practical go-to for many DRL systems because they directly provide training signal distributions. For robotics or sensitive data, consider generative replay variants or compressed coreset strategies.

Experience-focused methods diagram: short-term and long-term buffers, and optional generative model for replay.

Figure 10: Experience-focused methods: store or generate past experience and replay it to prevent forgetting.

3) Dynamic-focused methods

Dynamic-focused approaches learn models of environment dynamics. These models are helpful when dynamics shift over time:

  • Direct modeling: explicitly learn transition functions T(s’|s,a) and maintain a library of dynamics models (mixture models, CRP priors) to detect shifts and reuse models. Approaches like MOLe (mixture of experts) dynamically create specialized models for different contexts. Direct models are great for planning and sample efficiency, but maintaining many accurate dynamics models can be expensive.

  • Indirect modeling: learn compact latent variables or context embeddings that capture task properties without modeling full transitions. LILAC and similar latent-actor-critic methods let the agent infer an underlying latent that governs transitions and rewards. Indirect methods typically scale better and can be robust to partial observability.

Dynamics-focused methods shine when the environment’s rules change but tasks share structure. They are particularly useful in non-stationary robotics and simulation-to-real scenarios.

Dynamic-focused methods: either model transitions directly or learn latent representations that capture non-stationarity.

Figure 11: Dynamics-centered CRL. Direct models enable planning; latent models support compact adaptation and inference.

4) Reward-focused methods

The reward defines the task goal; shaping, reusing, or reconstructing reward functions can be a powerful transfer lever:

  • Reward shaping and potential-based transfers: modify new-task rewards using knowledge of prior task trajectories or potentials to accelerate learning.
  • Intrinsic rewards: curiosity and exploration bonuses (short-term and long-term) that push agents to learn skills useful across many tasks.
  • Latent reward decompositions: learn shared reward components that can be recombined for new tasks.

Reward-focused approaches are especially helpful in sparse reward settings and for enabling zero-shot or few-shot adaptation when task goals are related.

Reward-focused methods: shaping and intrinsic reward components that accelerate and guide learning across tasks.

Figure 12: Reward-focused CRL. Modifying reward signals is an effective way to transfer goal information.

Beyond the basics: task detection, offline RL, imitation, and more

CRL research increasingly includes:

  • Task (or change) detection: in task-agnostic scenarios you need to infer when the environment has changed. Detection methods use novelty detection, statistical tests on experience distributions, VAE reconstruction errors, or distributional distances (Wasserstein, KS tests).

  • Continual offline RL and imitation: learning from stored offline datasets or demonstrations is attractive when environment interaction is costly. Algorithms that selectively build replay buffers from offline data (e.g., model-based selection) can reduce forgetting without new exploration. Imitation learning and lifelong inverse RL expand reward-focused transfer.

  • Embodied continual agents: combining CRL with large pretrained models (LLMs or multimodal PTMs) is an emerging direction: use language or high-level planning from PTMs as a compact knowledge store and combine with fine-grained RL policies.

These directions help bridge lab benchmarks and real-world systems.

Timeline and representative milestones

Figure 5 in the survey gives a timeline of key developments from early modular and progressive approaches to recent hypernetwork, model-based, and large-scale hybrid methods. This historical arc shows that CRL is shifting from stability-only concerns toward richer combinations that emphasize transfer and scalability.

A timeline showing major CRL developments and milestones (methods, benchmarks, surveys).

Figure 13: A timeline of key developments in CRL illustrating the field’s rapid maturation.

Applications that matter

  • Robotics: modularity, replay, and policy decomposition are common. Benchmarks like Continual World are designed for robotic manipulation.
  • Games: Atari/Procgen and StarCraft provide rich visual and strategic settings for evaluating CRL approaches.
  • Language and embodied agents: CRL ideas are now applied to dialogue systems and embodied agents that must keep learning new tasks and topics.
  • Real-world control: data center cooling, fleet routing, finance — all benefit from continual adaptation under limited data.

For applied projects, the crucial design decisions are: Which knowledge type will you persist? How many resources can you afford for replay or models? Do you have access to task ids?

Practical recommendations

If you’re building or evaluating a CRL system, here are practical rules of thumb:

  • Start simple: finetuning + small replay buffer is a strong baseline. Compare against EWC and online-EWC variants.
  • Match the method to the constraint:
  • Memory-rich, task-aware → policy reuse or multi-head.
  • Memory-constrained, task-agnostic → compact replay + distillation or generative replay.
  • Changing dynamics → model-based / dynamics-focused methods.
  • Sparse rewards → reward shaping and intrinsic exploration bonuses.
  • Benchmark on at least two environments: one simple (e.g., MiniGrid) and one realistic (robot manipulation or a visual game) to capture transfer vs. scalability trade-offs.
  • Measure both performance and resource metrics: A_N, FG, FT, model size, sample efficiency, and wall-clock time.

Open challenges and promising directions

The survey highlights several open problems that are both intellectually interesting and practically urgent:

  • Task-free CRL: building agents that learn continuously in fully non-stationary environments with no task labels or clear boundaries.
  • Standardized evaluation: currently benchmarks and metrics vary widely. We need agreed-upon protocols that include resource metrics and privacy constraints.
  • Interpretable knowledge: moving beyond black-box parameters to knowledge structures that humans can inspect, reuse, and verify.
  • Large pre-trained models: two-way integrations — using PTMs (LLMs, vision-language models) as knowledge bases for CRL, and adapting PTMs themselves with CRL techniques (e.g., RLHF variants that are continual).
  • Continual embodied learning: robots and agents that must learn in-the-wild without resets — demands robust, scalable CRL solutions.

Wrap-up

Continual Reinforcement Learning is the bridge between single-task DRL and agents that can thrive over a lifetime of changing tasks. The survey we summarized organizes the field around a simple, practical question: what knowledge is stored and transferred? That question yields a taxonomy that maps cleanly to engineering choices — policy, experience, dynamics, and reward. Use this map to choose or design methods that match your application’s constraints.

If you take away one thing: there is no free lunch. CRL methods trade memory, computation, and adaptability against forgetting and transfer. The art (and science) is in choosing the right balance for your problem. The field is young and fast-moving — new benchmarks, generative replay advances, and hybrid PTM+RL systems will keep producing exciting developments. If you care about robots that learn forever, dialogue agents that stay current, or any system that must adapt continuously, CRL will be central to that future.

Further reading: the original survey (A Survey of Continual Reinforcement Learning) is an excellent entry point and contains a thorough bibliography of foundational and very recent works across the categories we discussed.