Can LLMs Master Lifelong Learning? Exploring MoE and Continuous Adaptation

Continual Lifelong Learning

Continual lifelong learning systems are adaptive algorithms that learn from a stream of information over time, where tasks are not predefined and the model must incorporate new data without catastrophic forgetting.

Key properties include knowledge retention, forward transfer, backward transfer, online learning, no task boundaries, and fixed model capacity.

LLM Properties

Large language models (LLMs) already satisfy many of these properties: pre‑training endows them with extensive world knowledge, small‑scale fine‑tuning rarely causes forgetting, while large‑scale continual training can lead to forgetting.

MoE Overview

Mixture of Experts (MoE) combines multiple expert models with a gating network that selects a subset of experts for each input, improving model capacity and efficiency.

Sparse MoE

Google Brain introduced sparse MoE, activating only a few experts during inference to reduce computation. Early experts tend to be selected more often, leading to the expert‑balancing problem.

Transformer MoE

GShard (2020) first integrated MoE into Transformer encoders/decoders, inserting MoE layers every other feed‑forward network. Subsequent works such as Switch Transformer, GLaM, and others scaled models to trillion‑parameter sizes.

PanGu‑Sigma (Huawei)

PanGu‑Sigma extends the Pangu‑Alpha model with MoE using Random Routing Experts (RRE). RRE employs a two‑layer gating design: the first layer assigns tasks to expert groups, and the second layer randomly balances load within each group.

Challenges of MoE

Structural complexity: Integrating MoE into Transformer architectures requires invasive modifications and extensive engineering effort.

Expert balancing: Data distribution is often skewed, and forcing uniform expert usage can hurt model performance.

Distributed communication: MoE introduces additional All‑to‑All communication, leading to significant network overhead in large‑scale training.