How Far Can Unsupervised RL for Large Models Go? A Systematic Answer from a Tsinghua Team

Paper and Problem

ICLR 2026 paper (https://arxiv.org/abs/2603.08660) studies the scaling limits of unsupervised reinforcement learning with intrinsic rewards (RLVR) for large language models.

Intrinsic Reward Taxonomy

Certainty‑based rewards: use the model’s own confidence (e.g., entropy, confidence score) on its inference trajectory as a proxy reward.

Ensemble‑based rewards: aggregate multiple roll‑outs (e.g., majority vote) to obtain a more reliable signal.

Unified Mechanism

All examined intrinsic‑reward methods share a “sharpening” operation that amplifies the model’s existing preference distribution. When the model’s prior (called “model prior” or “confidence‑correctness alignment”) is correct, sharpening improves performance; when the prior is wrong, sharpening accelerates collapse. This explains the characteristic “rise‑and‑fall” trajectory: rapid early gains followed by irreversible performance degradation.

Key Empirical Findings

Confidence‑correctness alignment determines success. Strong alignment leads to performance rise; weak alignment leads to rapid decline.

Small‑scale test‑time training (TTT) avoids collapse. With limited data, intrinsic rewards remain stable even when the initial model is completely wrong.

Model Collapse Step metric. Measures the number of training steps before collapse under intrinsic‑reward training. Later collapse correlates with higher RL‑trainability and outperforms traditional pass@k predictions.

External‑reward methods using generation‑verification asymmetry avoid the echo‑chamber effect. They show continuous improvement without collapse.

Large‑Scale Sweep

The authors evaluated 11 language models, 5 intrinsic‑reward variants, and four common hyper‑parameters (learning‑rate, batch size, reward scaling, and rollout count). Across all configurations the rise‑and‑fall pattern persisted, confirming its universality.

Model Collapse Step Details

Model Collapse Step is defined as the training step at which the proxy reward continues to increase while true performance drops to near‑random. Experiments show that models such as Qwen, known to be RL‑friendly, sustain many more steps before collapse than less suitable models. The metric requires no ground‑truth labels and predicts RL suitability more accurately than pass@k.

External Reward Approaches

Two categories are explored:

Unlabeled‑data‑driven rewards: extract signals from massive corpora, providing abundant, non‑depleting rewards.

Generation‑verification asymmetry: let the model generate answers and verify them with external tools (compilers, theorem provers, simulators). Preliminary experiments show monotonic improvement without collapse.

Implications

Intrinsic rewards are useful for early‑training or low‑data regimes but hit a hard scaling ceiling due to the sharpening mechanism. Scaling unsupervised RL likely requires external reward signals that are not confined to the model’s own confidence.

Resources

Implementation repository: https://github.com/PRIME-RL/TTRL/tree/urlvr-dev

Code example

来源：AI TIME 论道
本文
约3500字
，建议阅读
5
分钟
无监督 RLVR 的 scaling 上限在哪里？如果有上限，边界在哪里？