CRAM: Multimodal Models Forget? EMNLP 2026 Reveals Format Conflicts Mimic Forgetting
Nanjing University's CRAM method for multimodal continual instruction tuning isolates conflicting answer formats, reuses shared capabilities, and adaptively allocates LoRA experts via centroid routing and orthogonal constraints, revealing that many apparent forgetting errors are actually format mismatches rather than knowledge loss.
Introduction: The Paradox of Continual Multimodal Learning
Enabling models to continuously adapt to new data and tasks is a central challenge. However, after an update, performance on new tasks often improves while old tasks degrade. Researchers struggle to distinguish whether the model has truly lost knowledge or merely fails to express it under new output requirements. This distinction matters because the remediation strategies differ.
Key Observation: Format Conflicts Masquerade as Forgetting
The Nanjing University team conducted controlled experiments on the ArxivQA dataset. They kept the image and question semantics identical but varied the required answer format: option letter, option text, short answer, or yes/no. After fine-tuning on one format (e.g., "output option text"), accuracy on another format (e.g., yes/no) dropped by 50.2 percentage points relative to zero-shot. Analysis of the errors showed that 78.6% were "semantically correct but format non-compliant" — the model still knew the answer but persisted in the previously trained response style.
This demonstrates that many apparent forgetting errors are actually conflicts between answer conventions, not knowledge loss. However, completely isolating tasks would forfeit beneficial transfer. In a separate experiment, training on VizWiz-Caption improved Flickr30k scores by ~10 points over zero-shot, though not matching direct Flickr30k training. Thus, similar tasks share reusable capabilities while distribution gaps create new learning needs.
CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
CRAM addresses three intertwined problems: which requirements to isolate, which capabilities to share, and which gaps to fill with new parameters. It operates on a frozen multimodal backbone augmented with lightweight LoRA experts.
1. Semantic Grouping via Instruction Embeddings
When a new instruction arrives, CRAM uses a frozen CLIP text encoder to obtain its representation. It computes cosine similarity to existing group centroids and the difference between the highest and second-highest similarity. If the margin exceeds a threshold θ, the instruction joins the best-matching group; otherwise a new group is created. This ensures sharing only when the semantic match is unambiguous. Grouping is based on instruction semantics, not predefined format labels, so format compliance and reasoning are learned jointly while the routing mechanism adjusts their sharing scope.
2. Adaptive Rank Allocation via SVD Filtering
Instead of assigning a fixed LoRA rank per task, CRAM determines the required capacity per layer and attention module. It briefly warms up a temporary LoRA, performs singular value decomposition (SVD) on the update, and filters directions by two criteria: (a) Significance — singular value magnitude, discarding weak updates. (b) Independence — the component orthogonal to the historical input subspace of the group, computed as ||(I - U_h U_h^T) v|| where U_h is the orthonormal basis of historical directions. Only directions that are both strong and novel are retained; the number of retained directions sets the rank of the new expert. This yields per-task, per-layer, per-module rank allocations (see Figure 4 in the paper).
3. Stable Expert Routing with Visual Centroids
Even if old experts are preserved, a shifting router can misroute inputs, causing regression. CRAM assigns each expert a learnable visual centroid initialized from the mean of warm-up image features. During training, the centroid updates with the current task; after the task, it is frozen alongside the expert. At inference, image features from the frozen visual encoder are compared to centroids within the selected semantic group, and weights are assigned via a softmax over negative distances: closer centroids receive higher weight. This two-level routing (instruction semantics → group, visual features → expert) decouples format requirements from visual distribution shifts.
4. Orthogonal Penalty for Expert Complementarity
Multiple experts may redundantly learn similar updates. CRAM adds an orthogonal penalty to the generation loss that suppresses the new expert's response along historical input directions. The penalty strength grows with the current input's reliance on historical experts. This soft constraint encourages complementary learning throughout training, complementing the initialization-time SVD filtering which reduces capacity waste.
Results and Resources
The paper, "CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning," is accepted at EMNLP 2026. Resources:
Paper: arXiv:2606.02502 (https://arxiv.org/abs/2606.02502)
Code: GitHub - EMNLP2026-CRAM (https://github.com/LAMDA-CL/EMNLP2026-CRAM)
Toolkit: GitHub - Prism (https://github.com/LAMDA-CL/Prism)
CRAM unifies isolation of conflicting conventions, reuse of shared capabilities, and targeted parameter expansion, turning new skills into a foundation for future learning rather than a source of interference.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
