Why Does Large‑Model RL Training Narrow? Entropy Insights from ACL Paper

Reinforcement Learning with Verifiable Rewards (RLVR) is becoming a key technique for post‑training large language models, where correct math answers and passing code tests provide clear feedback signals. However, during RLVR training the policy entropy often drops rapidly, leading to output homogenization, narrowed sampling paths, and premature training stagnation.

The ACL 2026 Outstanding Paper "Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective" analyzes this entropy collapse from a token‑level perspective. By viewing RLVR training as dynamic adjustments of a reasoning tree, the authors show that each token update affects the effective branching factor of the tree, and that the overall entropy dynamics result from the accumulation of these token‑level changes.

Four factors determine the direction and magnitude of token‑level entropy change: (1) clipping strategy, (2) advantage signal (reward vs. penalty), (3) token generation probability (high‑ vs. low‑probability tokens), and (4) conditional entropy of the current context. Among these, advantage and token probability jointly dominate the entropy change, leading to four categories of token updates.

Existing entropy‑intervention methods such as DAPO/Clip‑Higher, Positive‑Reweighting, and Entropy‑Aware Advantage are reinterpreted as coarse‑grained adjustments of these factors. While they can mitigate entropy collapse to some extent, they lack fine‑grained control over each token’s entropy contribution.

To address this, the authors propose STEER (Stabilizing Token‑level Entropy‑change via Reweighting). STEER estimates the entropy change each token would cause in the current update and down‑weights tokens that would induce large entropy shifts, while leaving others largely untouched. This token‑wise “brake” avoids both rapid entropy collapse and uncontrolled entropy increase.

Experiments on multiple LLM families (Qwen, Llama, Mistral) and RLVR algorithms (GRPO, RLOO, OPO) demonstrate that STEER maintains policy entropy in a stable range even under minimal clipping constraints, and consistently improves performance on math reasoning and code tasks. For example, on Qwen2.5‑Math‑7B STEER raises the average score from 44.2 (GRPO) to 48.6, and on Qwen2.5‑Coder‑14B improves code editing from 42.6 to 45.1 and LiveCodeBench v5 from 29.3 to 31.8.

The paper concludes that dissecting entropy collapse to the token level transforms it from an opaque training curve symptom into an analyzable, explainable, and controllable dynamic, and that STEER offers a practical, fine‑grained mechanism to stabilize RLVR training.