Breaking the Forgetting Barrier: CaRE Scales Continual Learning to 300+ Tasks

Motivation

Human learners acquire new knowledge without catastrophic forgetting, but AI models suffer severe forgetting when updating parameters for new tasks. Class‑Incremental Learning (CIL) requires a model to learn new categories while preserving accuracy on historic ones. Existing CIL methods, even when built on large pre‑trained models, are typically evaluated on short task sequences of 5–20 tasks; performance degrades dramatically when the sequence length reaches hundreds of tasks, a gap that real‑world long‑running systems cannot tolerate.

Method – CaRE

CaRE (Scalable Continual Learner with efficient Bi‑Level Routing Mixture‑of‑Experts) embeds a Bi‑Level Routing Mixture‑of‑Experts (BR‑MoE) module into every Transformer block of a pre‑trained ViT. When a new task arrives, BR‑MoE learns only three sets of parameters: a class perceptron, a router network, and an expert adapter.

Two‑stage routing operates independently at each layer:

Dynamic router selection (first stage) : The [CLS] token is fed to all historic task class perceptrons; each task’s prediction entropy is computed. Tasks with the lowest entropy are deemed most semantically related, and the top‑M routers are selected without explicit task labels. This selection is performed per layer, enabling adaptive routing.

Dynamic expert routing (second stage) : The chosen routers generate weight scores for their associated expert adapters; the top‑K experts are fused with weighted averaging. An EMA‑updated shared expert continuously participates to preserve global cross‑task knowledge.

Because each layer executes the two‑stage logic independently, shallow layers retrieve generic visual features while deep layers retrieve highly task‑specific features.

New Benchmark – OmniBenchmark‑1K

OmniBenchmark‑1K is derived from OmniBenchmark‑V2 and contains 1,000 classes and ~190,000 images spanning 21 visual domains (birds, food, plants, actions, etc.). Overlap with ImageNet is removed, allowing seamless use of ImageNet‑pre‑trained models. Compared with benchmarks such as ImageNet‑R (≈200 classes), OmniBenchmark‑1K supports task sequences of 100, 200, or even 300+ tasks, providing a rigorous testbed for scalability.

Experimental Results

Long‑sequence performance : CaRE is compared against several strong continual‑learning baselines on OmniBenchmark‑1K with task sequences of 100, 151, 200, and up to 301 tasks. While many baselines collapse as the sequence length grows, CaRE maintains a stable learning trajectory and achieves a large margin over all baselines.

Short‑sequence performance : On classic short‑sequence benchmarks (ImageNet‑R, ImageNet‑A, ObjectNet), CaRE also ranks first, demonstrating a balanced trade‑off between plasticity and stability.

In‑Depth Analysis of Routing Activations

Shallow vs. deep layers : In shallow layers (e.g., Layer 3/6), a few experts are invoked frequently, extracting generic low‑level features (edges, textures). In deep layers (e.g., Layer 12), activation becomes sparse and highly task‑specific, matching the need for abstract semantic discrimination.

Knowledge crossing at test time : Even when evaluating early tasks, the model dynamically calls experts learned from later tasks, indicating that BR‑MoE enables flexible integration of global knowledge during inference.

Conclusion

CaRE is the first method empirically verified to run robustly on 300+ non‑overlapping tasks while remaining state‑of‑the‑art on standard short‑sequence benchmarks. The bi‑level routing mechanism unifies discriminative and comprehensive representation goals and injects dynamic knowledge retrieval into every network layer. The authors suggest that the two‑level routing idea could extend to cross‑modal continual learning (image, language, audio).

For full technical details and additional experiments, see the original paper “Scaling Continual Learning to 300+ Tasks with Bi‑Level Routing Mixture‑of‑Experts” (arXiv:2602.03473) and the open‑source code and data at https://github.com/LMMMEng/CaRE.