Oct 25, 2026 · Artificial Intelligence

Google Discovers Two 20-Year-Old Bugs in Softmax-Cross-Entropy Implementation

Google researchers identify two numerical bugs in the standard softmax-cross-entropy implementation used for two decades: Softmax Collapse zeros out gradients beyond a precision cliff, and Softmax Cancellation creates a noise floor in distillation; fixes using stable log-space computations eliminate Slingshot loss spikes in grokking and improve convergence across precisions.

Adam optimizerSlingshotcross-entropy

0 likes · 8 min read

Google Discovers Two 20-Year-Old Bugs in Softmax-Cross-Entropy Implementation

Machine Learning Algorithms & Natural Language Processing

Apr 4, 2026 · Artificial Intelligence

How Gram‑Newton‑Schulz Halves Muon Optimizer’s Compute Cost for Trillion‑Parameter Models

The article explains how the Muon optimizer’s expensive Newton‑Schulz orthogonalization is accelerated by the Gram‑Newton‑Schulz algorithm, which reduces end‑to‑end orthogonalization time by 40‑50%, achieves up to 2× speed‑up in large‑scale LLM training, and resolves numerical stability issues through a restart strategy and custom GPU kernels.

GPU KernelsGram Newton-SchulzLarge Language Models

0 likes · 9 min read

How Gram‑Newton‑Schulz Halves Muon Optimizer’s Compute Cost for Trillion‑Parameter Models