GROLE: Instance-Level Expert Routing for Incremental Learning in Oxygen AIIC Models

Introduction Large language models (LLMs) excel at zero‑shot generalisation but require task‑specific fine‑tuning for downstream performance, incurring prohibitive storage and compute costs and causing catastrophic forgetting when models are continually updated with new categories, attributes, or tasks.

Challenges in Incremental Learning (IL) Effective IL must (1) avoid catastrophic forgetting—new task updates must not degrade performance on previous tasks, and (2) promote knowledge transfer—shared structures should accelerate learning of future tasks. Existing LoRA‑based IL methods either regularise a single LoRA (e.g., O‑LoRA) and suffer severe forgetting under fuzzy task boundaries, or allocate independent LoRA experts per task and require explicit task IDs at inference, which is infeasible in real‑world, task‑agnostic streams.

Problem Formalisation The authors model a sequential data stream \(D\) of inputs \(x\) and labels \(y\) where each time step \(t\) corresponds to a (potentially undefined) task. The goal is to learn a parameter set \(W\) that minimises the cross‑entropy loss \(\ell(\cdot)\) on all observed tasks while keeping previously learned knowledge intact.

GROLE Framework

Stage 1 – Task‑Specific LoRA Experts

For each new task \(t\) a new LoRA expert \(i\) is instantiated, consisting of two low‑rank matrices. The expert is trained on the current task via standard supervised fine‑tuning (SFT) while all previously learned experts and the frozen pretrained backbone remain unchanged. After training, the full parameter set \(W\) (backbone + all LoRA experts) is frozen.

Stage 2 – Instance‑Level Selector

A lightweight selector network predicts, for every input instance, a set of merging weights \(\alpha\) that linearly combine the frozen experts. The selector is trained with reinforcement learning (RL) to avoid back‑propagating gradients through the large LLM. The RL reward is the cross‑entropy loss of the frozen SFT model, clipped to \([\delta_1,\delta_2]\) to stabilise training on easy or hard samples.

Dirichlet Sampling Strategy The selector outputs a concentration vector \(c\); a Dirichlet distribution \(\text{Dir}(c)\) is sampled to obtain \(\alpha\). This naturally respects the probability simplex and provides automatic exploration–exploitation balance: low concentration yields diverse samples (exploration), high concentration yields concentrated samples (exploitation).

Optimization Objective For each sample \(x\), the selector samples a group weight \(G\) and receives reward \(r\). The gradient‑free objective follows the Group‑Relative‑Performance Optimisation (GRPO) formulation, omitting the KL regulariser because the selector is randomly initialised.

Experiments

Benchmarks: (1) Standard CL (AG News, Amazon, Yelp, DBpedia, Yahoo Answers – 4 tasks), (2) Large Number of Tasks (12 tasks covering CL, GLUE, SuperGLUE, IMDB). Data‑splitting strategies include Srd‑1 (sequential, no overlap), fragment‑shuffle (4 fragments per task), and Dir‑0.3 (Dirichlet‑sampled task assignment for highly fuzzy boundaries). Evaluation metric is Average Accuracy (AA) across all tasks, capturing both backward (j < t) and forward (j > t) transfer.

Baselines: non‑incremental upper bounds (PerTaskLoRA, MTL) and incremental methods (Replay, SeqLoRA, IncLoRA, O‑LoRA, MultiLoRA, MoELoRA, MTL‑LoRA). GROLE outperforms the best baselines by 4.32 % on Standard CL and 9.47 % on the Large‑Task benchmark, even surpassing non‑incremental upper bounds, demonstrating that instance‑level dynamic routing can exceed static multi‑task training.

Negative‑Transfer Analysis In fuzzy‑boundary settings (Srd‑4, Dir‑0.3), orthogonal‑constraint methods (O‑LoRA) suffer severe degradation because strict parameter isolation prevents shared feature learning. MultiLoRA exhibits instability due to implicit competition among modules. GROLE’s adaptive merging mitigates negative transfer, with performance gains increasing as task overlap grows.

OOD Generalisation Using AG News as an out‑of‑distribution task, GROLE improves OOD accuracy by 3.60 % over the strongest baseline and remains stable across all splitting strategies, indicating strong adaptability to unseen task types.

Ablation Studies

Group size sensitivity: varying \(G\) ∈ {4, 8, 16, 24, 32} changes performance by only ~0.6 %, showing robustness to this hyper‑parameter.

Top‑k expert activation: activating only the top‑1 expert already reaches near‑full performance; increasing the number of activated experts yields steady improvements, confirming the benefit of multi‑expert fusion.

Sampling strategy: Dirichlet sampling outperforms Gaussian noise by 0.81 % (Standard CL) and 1.85 % (Large‑Task), highlighting the advantage of simplex‑constrained exploration.

Training efficiency: Stage 2 selector training consumes less than half the time of Stage 1 expert training and supports batched instance‑level inference.

Conclusion GROLE provides a modular, scalable solution for industrial‑scale incremental learning without task identifiers. By freezing a growing LoRA expert pool and employing a reinforcement‑learning‑driven instance‑level selector, it eliminates catastrophic forgetting while enabling strong forward knowledge transfer, achieving a superior balance between stability and plasticity in open‑world environments.