Octopus: History‑Free Gradient Orthogonalization for Continual Learning (CVPR 2026)

Continual learning for multimodal large language models (MLLMs) faces catastrophic forgetting when new tasks overwrite knowledge from previous tasks. Existing rehearsal‑based methods require storing historic data, raising privacy and storage concerns; architecture‑based approaches add task‑specific parameters, hurting scalability; regularization‑based techniques struggle to balance plasticity and stability.

Octopus tackles this dilemma by proposing two core innovations. First, it introduces History‑Free Gradient Orthogonalization (HiFGO), a gradient‑level orthogonal constraint that ensures updates for a new task are orthogonal to the “key gradient space” of previous tasks. To approximate the unavailable historic gradients, Octopus defines GPWC (Gradients of Previous parameters Within Current data distribution), which computes gradients of old parameters on the current task’s data, theoretically equivalent to projecting the old task’s Hessian onto the new data manifold.

Second, Octopus adopts a two‑stage finetuning strategy. In the initial stage, the model freely learns the new task without constraints, maximizing plasticity. In the second stage, HiFGO is applied to pull the parameters back onto a safe manifold that preserves old knowledge while retaining high performance on the new task. This “learn‑then‑consolidate” process mitigates the plasticity‑stability trade‑off.

Empirical evaluation on the UCIT benchmark—comprising six diverse multimodal tasks—shows that Octopus achieves average (Avg) and final (Last) scores of 71.08% and 71.01%, surpassing previous SOTA methods by 2.14% and 6.82% respectively. It also matches or exceeds the theoretical upper bound set by full‑data multi‑task training on tasks such as ArXivQA and CLEVR‑Math, and outperforms rehearsal‑based baselines despite using no historic data.

Beyond accuracy, Octopus demonstrates positive backward transfer (BWT = +0.41), whereas conventional orthogonal regularization yields negative transfer (BWT = –2.51). Hessian quadratic‑form analysis confirms that HiFGO reduces parameter updates in high‑curvature regions of old tasks, effectively decoupling knowledge.

Robustness experiments reveal that Octopus maintains stable performance across a wide hyper‑parameter range and is insensitive to task order permutations, indicating strong “sequence immunity.” Inference analysis shows that, unlike MoE‑based experts that add parameters per task, Octopus retains a single LoRA module, incurring zero additional inference overhead—making it suitable for edge deployment.

Overall, Octopus provides a privacy‑preserving, parameter‑efficient, and high‑performing solution for lifelong learning in multimodal LLMs, advancing both academic understanding and practical deployment of continual learning systems.