Why Large Models Get More Stable with More Edits: Unveiling Lifelong Normalization

Large language models store vast world knowledge, but facts become outdated over time. Model editing offers a low‑cost way to update specific knowledge, yet lifelong model editing (LME) faces two amplified challenges when edits scale to thousands or millions: catastrophic forgetting and model collapse.

Recent methods such as ULTRAEDIT and RLEdit remain stable at large scale because they all incorporate a component called Lifelong Normalization (LN). Ablation experiments reveal that removing LN causes a dramatic drop in long‑run performance, indicating that LN is the pillar of stability, although its inner workings have been a black box.

What LN Does

At each edit step the editor extracts two vectors: (i) the hidden‑state input of the target layer (e.g., the down_proj layer) and (ii) the loss gradient with respect to that layer (the “value gradient”). LN maintains online moving averages of the mean and variance of these value‑gradient vectors, normalizes the current gradient using these statistics, and then feeds the normalized gradient into a ridge‑regression solver to obtain a closed‑form parameter update.

Because the model parameters change after every edit, the gradient distribution drifts. LN treats this drift as a dynamic distribution and tracks it via a recursive Bayesian estimator (Normal‑Inverse‑Wishart prior), providing online estimates of the mean and covariance.

Why Edits Become More Stable

Two theoretical insights explain the observed “positive cumulative effect”:

Under mild regularity conditions, if each parameter update is controlled, the drift of the gradient distribution is bounded. Consequently, the moving‑average estimates become more accurate as more samples accumulate (Theorem 3.5 and Theorem 3.6), reducing estimation error at a rate proportional to the number of processed samples.

Accurate estimates lead to parameter updates with bounded norm and weak interference between successive steps (Theorem 3.8). This prevents model collapse and mitigates catastrophic forgetting. Early edits therefore act as extra samples that refine LN’s statistics, making later normalizations more reliable.

This explains the counter‑intuitive observation that a model that has undergone many prior edits actually performs better on subsequent edits.

StableEdit: Strengthening LN

Two weaknesses of existing LN‑based editors are identified: (1) the statistics start from a noisy state, and (2) the per‑dimension normalization discards inter‑dimensional correlations. StableEdit addresses them with:

Warm‑up phase : run a small number of dummy edits before the target edit stream to pre‑populate LN’s moving statistics, providing a more reliable starting point.

Full whitening : replace per‑dimension scaling with full covariance whitening, preserving the covariance structure of the value gradients and yielding cleaner geometric updates.

These enhancements add only a covariance‑decomposition cost and, on 7‑8 B models, are even cheaper than baselines such as RLEdit or AlphaEdit.

Experiments

Evaluations on standard‑scale benchmarks (ZsRE, FEVER, ULTRAEDITBENCH, WikiBigEdit ≈ 50 K edits) and extreme‑scale benchmarks (ULTRAEDITBENCH 2 M edits) show that StableEdit consistently outperforms ULTRAEDIT, RLEdit and other baselines in efficacy, generalization and specificity, while preserving performance on GLUE tasks and maintaining hidden‑state distributions (UMAP visualizations show near‑identical clusters before and after editing).

Ablation studies confirm that removing LN leads to severe degradation, and that both warm‑up and full whitening contribute to stability, with LN being the foundational component. Additional component‑wise ablations demonstrate that warm‑up improves the initial statistical estimate, while full whitening improves the geometry of the update direction.

Conclusions

The paper provides the first theoretical explanation of LN as an online Bayesian tracker of dynamic gradient distributions. Accurate statistical tracking yields bounded‑norm, weakly interfering updates that simultaneously alleviate catastrophic forgetting and model collapse. Building on this insight, StableEdit incorporates warm‑up and full whitening to achieve robust, million‑scale lifelong editing.

Highlights

LN is the key mechanism for stability in lifelong model editing; removing it causes severe degradation.

Early edits positively reinforce later edits by providing more reliable statistical priors.

Combining LN with ridge regression yields bounded‑norm, low‑interference updates, mitigating forgetting and collapse.