How Skipping 50% of Gradient Updates Supercharges LLM Training (SkipUpdate & Magma)

SkipUpdate: The Simplest Effective Strategy

In deep learning it is commonly believed that more complete gradient updates lead to better models. Adaptive optimizers such as Adam and RMSProp are popular because they exploit full gradient information. The paper from Google and Northwestern challenges this intuition.

Randomly dropping 50% of parameter updates does not harm training; it actually improves model performance.

Core Algorithm

for each parameter block b:
    m_t^(b) ~ Bernoulli(0.5)  # 50% mask
    θ_{t+1}^(b) = θ_t^(b) - s_t^(b) * m_t^(b) * Δ_t^(b)

Key Design

Random mask : each block is skipped with 50% probability.

Momentum preservation : even when a block is skipped, its momentum estimate is updated densely.

Unbiased correction : scaling factor s_t = 1/p = 2 guarantees an unbiased update.

Why It Works: Theoretical Insight

The authors prove (Proposition 1) that the random mask introduces a curvature‑dependent geometric regularization term into the expected loss. This term penalizes updates in high‑curvature directions, smooths the optimization trajectory, and implicitly reproduces Sharpness‑Aware Minimization (SAM) without extra computation.

Transformers have a block‑diagonal Hessian structure, so block‑level masking aligns with the dominant curvature interactions.

Magma: From Random Masking to Intelligent Masking

SkipUpdate treats all parameter blocks uniformly, but Transformer parameters are heterogeneous. Magma (Momentum‑aligned Gradient Masking) refines the mask by aligning it with the direction of the momentum.

Core Innovation: Momentum‑Gradient Alignment

Updates whose gradient aligns with momentum are likely signal; conflicting directions are likely noise.

The alignment score is computed for each block; a scaling factor close to 1 retains the update, while a score near 0 suppresses it. A temperature parameter τ = 2 controls sensitivity.

Key Advantages

✅ Zero additional overhead – only a scalar multiplication per block.

✅ Plug‑and‑play – can wrap any adaptive optimizer.

✅ Theoretical guarantee – retains geometric regularization while improving stability.

Experimental Results: Comprehensive Superiority

4.1 Llama 2 Pre‑training (C4 dataset)

Across model sizes from 60 M to 1 B parameters, SkipUpdate consistently beats state‑of‑the‑art optimizers (including Muon). Magma further reduces validation perplexity, achieving a 19 % drop for the 1 B model compared to Adam.

4.2 MoE Architecture: Complex Optimization Testbed

In a Nano MoE setting, Magma combined with Muon yields the best performance, surpassing the Cautious Optimizer (which also uses momentum‑gradient alignment but lacks random masking).

4.3 Heavy‑tailed Noise Environment

LLM training gradients exhibit heavy‑tailed noise. Under controlled experiments, Magma dramatically outperforms Adam in heavy‑tailed settings while maintaining comparable performance under light‑tailed noise.

4.4 Heterogeneous Quadratic Functions: Theory Validation

On synthetic heterogeneous Hessian structures (mimicking Transformer spectra), Magma converges faster and reaches lower final loss than AdamW. In a CNN (ResNet‑50) setting, Magma shows no advantage, confirming its benefit is specific to Transformer‑like geometry.

Theoretical Analysis: Why Magma Is Effective

Convergence Guarantee (Theorem 6)

The paper establishes a non‑convex convergence rate, showing that the alignment score simultaneously influences descent efficiency and noise level. Selective suppression of high‑curvature/high‑variance blocks expands the stable learning‑rate range, explaining Magma’s robustness to learning‑rate variations.

Comparison with Existing Methods

Cautious Optimizer : deterministic mask, no geometric regularization, no extra cost.

SAM : adversarial perturbation, provides geometric regularization, doubles gradient computation.

GaLore : subspace projection, no geometric regularization, saves memory.

Magma : random mask + alignment modulation, provides geometric regularization, zero extra cost.

Ablation Studies

Mask Component Effects

Attention‑only mask: modest improvement.

Attention + MLP mask: best performance.

All‑layer mask: slightly worse than selective masking.

Granularity Choice

Element/Row/Column/Block levels yield similar gains.

Block‑level recommended for optimal memory efficiency.

Learning‑Rate Robustness

Adam and C‑Adam collapse outside a narrow 0.001‑0.003 window, whereas Magma remains stable across 0.0001‑0.05, reducing the need for extensive hyper‑parameter tuning.

Reference

https://arxiv.org/pdf/2602.13517