Machine Heart
Nov 20, 2026 · Artificial Intelligence
Massive Activations in LLMs Function as Gradient Regulators, Study Reveals
Researchers from Tsinghua and Peking University discover that massive activations in transformer models act as adaptive gradient regulators, with attention sinks creating gradient sinks that RMSNorm suppresses, revealing a backward-propagation mechanism linking attention concentration to activation scaling.
Attention SinkBackpropagationGradient Sink
0 likes · 11 min read
