Why PEFT Evaluation Must Go Beyond Downstream Scores: Quantifying General Capability Loss

Parameter‑efficient fine‑tuning (PEFT), exemplified by LoRA, updates only a small subset of model parameters, reducing training cost and enabling rapid deployment across tasks. Traditional PEFT evaluations focus almost exclusively on downstream task accuracy, ignoring what pretrained capabilities the model may forget during adaptation.

PEFT‑Arena, introduced by researchers from The Chinese University of Hong Kong, Westlake University, and the Max Planck Institute, reframes this problem as a classic stability‑plasticity dilemma. It proposes a dual‑axis benchmark that measures (1) target‑domain performance (plasticity) and (2) retention of pretrained general abilities (stability). Experiments use Qwen2.5‑7B and Llama3.2‑3B‑Instruct, fine‑tuned on mathematical and medical reasoning tasks via supervised fine‑tuning (SFT) and reinforcement learning with validation rewards (RLVR). General‑ability retention is assessed with IFEval, Natural Questions, BBH, and related tasks.

The two‑dimensional evaluation plot places ideal methods in the upper‑right corner, indicating high target performance with minimal loss of general abilities. Most PEFT methods exhibit a clear trade‑off: full‑parameter fine‑tuning achieves strong target scores but severely degrades general abilities; LoRA‑type low‑rank methods are more conservative yet still incur noticeable forgetting; PiSSA can boost target scores dramatically but at the cost of severe general‑ability loss; VeRA maintains stability but offers limited target gains. Orthogonal fine‑tuning (OFT) often lands on a more competitive frontier, achieving comparable target improvements while preserving more general capability.

RLVR training generally exhibits weaker forgetting than SFT and can sometimes improve both target and general scores, though prolonged RLVR can degrade high‑k sampling metrics (e.g., pass@64) despite stable pass@1 performance, highlighting the need for path‑level diagnostics.

To explain these observations, PEFT‑Arena analyzes weight‑space geometry. By decomposing pretrained weight matrices into singular‑vector bases, the authors define two views: a retention‑profile measuring how much of the original singular structure is preserved, and an update‑energy profile indicating where updates concentrate. Low‑rank methods like LoRA produce concentrated updates; PiSSA interacts strongly with leading singular directions, causing larger structural perturbations; OFT’s orthogonal parameterization tends to preserve the original spectral geometry.

The study introduces Capability‑Conditioned Drift (CSD) to quantify how weight updates affect activations on general versus target data distributions. Experiments show that higher CSD on general data correlates with forgetting, while target‑domain CSD does not reliably predict target scores.

Activation‑space analysis further clarifies forgetting. Three geometric metrics are used: Procrustes residual (unalignable structural change after optimal orthogonal alignment), Gram matrix distortion (change in pairwise similarity), and Centered Kernel Alignment (CKA) (overall representation similarity). Larger Procrustes residuals and Gram distortions correspond to greater forgetting, whereas higher CKA indicates better retention. OFT maintains geometric structure despite moving representations, whereas PiSSA induces non‑isometric distortions linked to severe forgetting.

Beyond endpoint comparison, PEFT‑Arena examines the fine‑tuning trajectory via parameter interpolation between the base model and the final checkpoint. Many SFT settings reveal that intermediate interpolation points retain most target gains while restoring a substantial portion of general ability, demonstrating that the final checkpoint is not always the optimal trade‑off point.

Building on this, the authors explore path‑wise rewinding: applying controlled rollback along the appropriate parameter‑space trajectory (e.g., along the Cayley generator for OFT) can improve the target‑retention balance without retraining. Experiments illustrate this effect for OFT and, in the appendix, for additive PEFT methods such as LoRA and MiSS.

Overall, PEFT‑Arena expands PEFT evaluation from a single downstream accuracy figure to a two‑dimensional assessment of target adaptation and pretrained capability preservation, provides geometric explanations for forgetting, and offers practical tools—interpolation analysis and pathwise rewinding—to locate better trade‑off points.