Google Discovers Two 20-Year-Old Bugs in Softmax-Cross-Entropy Implementation

The standard softmax plus cross-entropy implementation — used in JAX, PyTorch, and TensorFlow — has followed the same textbook recipe since 2002: subtract the maximum logit to avoid overflow, then exponentiate and normalize. Google researchers have now uncovered two numerical bugs hiding in this ubiquitous code.

Two Bugs: Precision Cliff and Cancellation Error

First bug: Softmax Collapse. As the model becomes increasingly confident on the correct class (measured by logit margin m ), the target logit's gradient should shrink exponentially but remain non-zero. In the standard implementation, once m crosses a "precision cliff" — approximately 5.5 for bfloat16, 16.6 for fp32, 36.7 for fp64 — this gradient is rounded to exactly zero.

Second bug: Softmax Cancellation, which especially hurts distillation. When student logits approach teacher logits, the standard implementation computes probabilities q and p separately and subtracts them. Subtracting two nearly equal probabilities loses precision; worse, the forward pass uses softmax while the backward pass uses log-softmax, so the formula paths differ. Even when student and teacher are identical, the gradient is not zero, leaving an irreducible noise floor.

Fix: A One-Line Core Change

The fixes are surprisingly simple:

Never add 1 and e^-m directly. Set the maximum logit to exactly 0 and replace log(1+e^-m) with the numerically stable softplus(- m ). Automatic differentiation then yields "target gradient = negative sum of non-target gradients", which is naturally zero-sum.

Compute q − p directly in log space. Use expm1 and log1p to avoid any "subtract 1" operations. When student equals teacher, the gradient becomes exactly zero.

No precision changes, no internal casting, a few lines of drop-in replacement with virtually no overhead.

Slingshot's True Cause: A Rounding-Error Cycle

In grokking training, networks memorize the training set quickly but generalize late, with mysterious loss spikes — the Slingshot phenomenon recorded by Thilak et al. (2022). The paper provides a complete causal chain:

Margin crosses precision cliff → target gradient zeroed → parameter gradients nearly vanish → Adam's second moment √v̂ decays exponentially → weight decay pulls individual samples back below the cliff → target gradient suddenly recovers while Adam divides by a tiny √v̂+ε → a huge parameter update → loss spike → cycle repeats.

The fixed cross-entropy lets gradients cross the cliff smoothly, eliminating spikes entirely. Experiments show: with weight decay, the fixed version achieves 100±0% grokking success across 5 seeds (standard: 94±5%); without weight decay, standard crashes at ~1,250 steps, test accuracy collapsing from 61% to 23%, while the fixed version reaches 85%.

A Neat Byproduct: m* ≈ log(1/ε)

After the fix, margin stops growing at around 18.4 — not due to precision limits, but because of a balance between weight decay and Adam's normalized step length. When gradient |g| shrinks far below ε, Adam's normalized step decays with e^-m and balances weight decay λθ; solving gives m * ≈ log(1/ε). This explains why the default ε=10^-8 yields an equilibrium at 18.4, just above the fp32 cliff at 16.6 — the standard implementation always hits the cliff before reaching equilibrium.

LaProp experiments in fp32 across many ε values confirm the analytical prediction; with ε=0, margin grows until the gradient itself underflows (m≈81).

Experiments: Fixed Version Passes All Precisions

On the a+b mod 31 grokking benchmark, the standard implementation fails at every precision cliff: bfloat16 stalls at m≈5.5, fp32 oscillates at 16.6, and even fp64 crashes at its cliff (36.7) when ε is lowered to 10^-16. The fixed version converges smoothly to 100% test accuracy in all three precisions.

Compared to two recent improvements: Zero-sum Projection only restores a -1/K gradient to the target logit, margin sticks near 18, and accuracy drops below 50% for ε≤10^-12; StableMax replaces softmax with a linear alternative, margin explodes to 10³–10⁴, and accuracy lags across the board.

Generalization holds: on four distinct grokking tasks from the literature, fp32 fixed matches fp64 standard, while fp32 standard fails at the cliff every time.

Distillation experiment (student initialized with teacher weights): the fixed version yields exact zero gradient; standard leaves a noise floor.

A Fix for Softmax and Cross-Entropy
https://arxiv.org/abs/2610.31642