Tagged articles

entropy collapse

3 articles · Page 1 of 1

Machine Learning Algorithms & Natural Language Processing

Jul 24, 2026 · Artificial Intelligence

Why Large-Model RL Training Narrows Over Time? ACL 2026 Paper Reveals Entropy Collapse

The article analyzes why reinforcement learning with verifiable rewards (RLVR) for large models experiences rapid policy‑entropy collapse, breaks the phenomenon down to token‑level entropy changes driven by clipping, advantage, token probability and conditional entropy, and introduces STEER, a token‑wise reweighting scheme that stabilizes entropy and yields consistent performance gains on math and code benchmarks.

Large Language ModelsMachine learningRLVR

0 likes · 14 min read

Why Large-Model RL Training Narrows Over Time? ACL 2026 Paper Reveals Entropy Collapse

Machine Heart

Jul 15, 2026 · Artificial Intelligence

Why Does Large‑Model RL Training Narrow? Entropy Insights from ACL Paper

Large‑model reinforcement learning with verifiable rewards often suffers entropy collapse, causing exploration to shrink; this article dissects the phenomenon at the token level, identifies four influencing factors, critiques existing entropy interventions, and introduces STEER—a token‑wise reweighting scheme that stabilizes entropy dynamics and yields consistent gains on math reasoning and coding benchmarks.

LLMRLVRSTEER

0 likes · 12 min read

Why Does Large‑Model RL Training Narrow? Entropy Insights from ACL Paper

Machine Heart

Jun 21, 2026 · Artificial Intelligence

Why Post‑Training Makes Large Reasoning Models Overconfident and How LED Restores Exploration

The paper reveals that reinforcement‑learning post‑training flattens the entropy of the final layer in large reasoning models, making higher sampling temperatures ineffective, and introduces Latent Exploration Decoding (LED) to recover exploration from intermediate layers, yielding consistent pass@k gains without extra training.

LED methodRL post-trainingentropy collapse

0 likes · 13 min read

Why Post‑Training Makes Large Reasoning Models Overconfident and How LED Restores Exploration