University of Alberta's FAME Framework Tackles Catastrophic Forgetting in Continuous RL

From Empirical Fixes to a Principled Analytical Framework

Traditional reinforcement learning assumes a fixed environment with stationary reward functions and state transitions. Real-world agents, however, need both plasticity — rapid adaptation to new tasks — and stability — retention of previously acquired skills. These two capabilities are in tension: learning new tasks often overwrites old knowledge (catastrophic forgetting), while old knowledge can hinder learning when tasks differ greatly. Existing continual RL methods (experience replay, parameter regularization, network expansion, policy distillation) are empirical fixes lacking a unifying theoretical foundation.

Theoretical Foundations: Defining Environment Distance and Catastrophic Forgetting

The Alberta team first formalizes two core concepts. (1) Environment distance : the difference between two Markov decision processes is measured by the distance between their optimal value functions or optimal policies, capturing both reward and transition differences in a mathematically clean way. (2) Catastrophic forgetting in sequential decision-making : unlike supervised learning, the data distribution in RL depends on the policy. The authors define forgetting via the value or policy function under the state-visitation distribution of past tasks, weighting states by how often they were actually visited — focusing on the states and actions that truly mattered in previous tasks rather than the entire state space. This turns the knowledge-integration step into an explicit catastrophic-forgetting minimization optimization problem.

FAME: Dual-System Learning Inspired by Hippocampus and Cortex

FAME (Principled Fast and Meta Knowledge Learners) mirrors the hippocampal-cortical memory system. The Fast Learner (hippocampus) handles knowledge transfer for rapid adaptation; the Meta Learner (cortex) consolidates new knowledge into long-term memory. Sun Ke explains that a single model cannot fit infinitely many shifting data distributions — hence two specialized systems.

Adaptive Warm-Start in the Fast Learner

When a new task arrives, the Fast Learner must choose a starting point: inherit the meta-learned policy, fine-tune the previous task's policy, or initialize randomly. FAME frames this as a one-vs-all statistical hypothesis test . If historical knowledge helps, use meta-learned initialization; if the new task resembles the immediate predecessor, fine-tune; if it is entirely novel, random initialization is preferred. The paper includes a figure showing the proportion of optimal candidate strategies selected across different environment shifts.

Knowledge Consolidation in the Meta Learner

After the Fast Learner masters a new task, the Meta Learner integrates it by minimizing the catastrophic-forgetting objective defined earlier. For discrete actions this reduces to an incremental maximum-likelihood objective ; for continuous actions it corresponds to minimizing KL divergence or Wasserstein distance . This mathematical equivalence bridges continual RL and multi-task learning.

Experimental Evaluation

Experiments span MinAtar, Atari, and Meta-World (robotic manipulation), covering discrete and continuous actions, value-based and policy-based methods (DQN, PPO, SAC). FAME is compared against Reset, Fine-tuning, Average, PackNet, ProgressiveNet, and CompoNet. On average performance, forward transfer, and forgetting metrics, FAME outperforms all baselines, with advantages growing on later tasks. Ablation studies confirm that both adaptive warm-start and proper regularization strength/memory buffer size are necessary — removing either degrades performance.

Continuous Learning: Academic Niche or Inevitable Hurdle for Superintelligence?

The authors argue that scaling laws and periodic retraining cannot scale to superintelligence; the data-search–pretrain–finetune–post-train pipeline becomes infeasible. Zhang Hongming notes every major AI breakthrough (DQN, AlphaZero, AlphaProof, RLHF, GRPO) has RL at its core, and breaking capability ceilings requires algorithmic innovation, not just human-engineered data. The industry consensus (Hassabis, Liang Wenfeng, Ilya Sutskever) identifies continuous learning and long-term memory as the unsolved core barrier. Self-evolution (DeepSeek R1-Zero, etc.) is essentially RL: interact, get feedback, improve, repeat.

Embodied Intelligence and Unsupervised Exploration

World models correspond to model-based RL; vision-language-action (VLA) models are model-free. The sim-to-real gap in motor control has narrowed significantly — RL training in simulation followed by real-world fine-tuning works well. Berkeley's SERL system achieves 100% success rates on real robots . On unsupervised exploration, Sun Ke states: "Unsupervised is always correct, but not necessarily the most efficient" — good representations matter, but downstream performance ultimately requires task-specific supervised signals.

Author Backgrounds

Zhang Hongming and Sun Ke met at Peking University, then joined Richard Sutton's RLAI lab at the University of Alberta. Zhang (advised by Martin Müller, AlphaGo team mentor) is now an assistant researcher at the Chinese Academy of Sciences' Institute of Automation, applying RL to chip design optimization. Sun is a postdoc at the University of Pennsylvania, planning to return to China and collaborate with industry while pursuing fundamental scientific questions.

Limitations and Conclusion

FAME's Q-value-based forgetting metric is sensitive to reward scale (the policy version is more stable); knowledge-transfer mechanisms are still evolving (e.g., context embeddings). The framework assumes existing policies and cannot solve tasks beyond all current policies' capabilities. Nevertheless, it demonstrates that continual learning can move from empirical patching to a principled, mathematically grounded science . The authors' recurring message: superintelligence cannot be built by stacking data and retraining — agents must learn, use, and not forget , just like humans.