DeepHub IMBA

Nov 6, 2026 · Artificial Intelligence

Soft-MTP: Fixing Gradient Noise in Multi-Token Prediction via Cross-Entropy Decomposition

Soft-MTP decomposes the cross-entropy gradient in multi-token prediction into a distribution-alignment term and a residual term, then cancels the residual from gradients flowing to MTP heads and the shared backbone while preserving cross-entropy training on the shared output projection, improving language modeling and speculative decoding without extra parameters or full KL computation.