Why Large-Model RL Training Narrows Over Time? ACL 2026 Paper Reveals Entropy Collapse

Reinforcement Learning with Verifiable Rewards (RLVR) has become a key technique for post‑training large language models, where correct answers to math problems or passing code tests provide clear feedback. However, during RLVR training the policy entropy often drops quickly, causing the model’s outputs to become homogeneous and the sampling space to narrow, which can stall training.

Entropy Collapse as a Tree‑Pruning Process

The authors view a model’s generation as a dynamic inference tree: each token selection expands a branch, and the token‑level entropy reflects the number of effective branches at a node. Policy entropy decline corresponds to aggressive pruning of branches, while token‑level entropy increase reflects branch expansion. When pruning dominates, the exploration space contracts, leading to entropy collapse.

Token‑Level Entropy Change Decomposition

By deriving an approximate expression for token‑level entropy change after one update, the paper isolates four factors (after removing learning‑rate and length‑normalization terms):

Clipping strategy : whether the token’s importance‑sampling ratio is clipped.

Advantage signal : whether the token comes from a positive or negative sample, i.e., if the update rewards or penalizes it.

Token probability : whether the token is originally high‑probability (easy) or low‑probability (rare).

Conditional entropy : the remaining uncertainty of the model given the current context.

Empirical analysis shows that the direction of entropy change is mainly determined by the advantage signal and the token’s generation probability. Combining these two dimensions yields four token‑update categories, which are illustrated by mapping each case back onto the inference tree.

Re‑examining Existing Entropy‑Intervention Methods

From the token‑level perspective, several prior entropy‑control techniques can be interpreted as coarse adjustments to the above factors:

DAPO / Clip‑Higher : relaxes clipping for low‑probability but correct tokens, encouraging rare yet correct paths.

Positive‑Reweighting : reduces the weight of high‑probability positive tokens and boosts low‑probability positives, aiming to preserve diverse correct branches.

Entropy‑Aware Advantage : assigns larger advantage to high‑entropy tokens under the intuition that uncertain tokens should be explored more.

The authors point out that these methods are heuristic and operate at a coarse granularity; they modify clipping thresholds or advantage weights without directly estimating each token’s actual entropy impact, limiting their ability to precisely control training dynamics.

STEER: Stabilizing Token‑Level Entropy Change via Reweighting

To address the coarse nature of existing interventions, the paper proposes STEER (Stabilizing Token‑level Entropy‑changE via Reweighting). STEER first estimates the entropy change each token would cause in the current update, then inversely reweights the training signal: tokens that would cause large entropy swings are down‑weighted, while those with modest impact retain near‑original weights. This token‑wise “brake” prevents sudden entropy drops without blindly increasing overall entropy.

Experiments on a range of math and code benchmarks demonstrate that STEER keeps policy entropy in a stable band even under minimal clipping constraints, avoiding both rapid collapse and uncontrolled entropy growth. Performance gains are consistent: on six math benchmarks, Qwen2.5‑Math‑7B improves from 44.2 (GRPO) to 48.6; on code tasks, Qwen2.5‑Coder‑14B rises from 42.6 to 45.1 on the primary metric and from 29.3 to 31.8 on LiveCodeBench v5. Additional tests across model families (Qwen, Llama, Mistral) and RLVR algorithms (GRPO, RLOO, OPO) confirm that STEER’s benefits are not tied to a specific architecture or optimizer.

Conclusion

The work’s main contribution is decomposing RLVR entropy collapse to the token‑update level, showing how individual token dynamics aggregate to shape the whole inference tree. By directly controlling token‑wise entropy change, STEER offers a finer‑grained, theoretically grounded remedy that both stabilizes training dynamics and yields measurable improvements on downstream reasoning and coding tasks.