Unsupervised RL for Large Models: How Far Can It Scale? Tsinghua’s Systematic Study

Problem and Motivation

Reinforcement learning with verifiable rewards (RLVR) is used in large language models to push inference performance, but pure supervised training is unsustainable because human annotation costs grow exponentially, especially in expert domains. Unsupervised RLVR (UR‑RLVR) aims to let models keep improving without any human labels, raising the question of its scaling limits.

Methodology

The investigation consisted of five actions:

Unified theoretical framework : mapped diverse intrinsic‑reward methods to a single mechanism that sharpens the model’s initial distribution and derived convergence bounds.

Large‑scale empirical study : trained 11 models with 5 intrinsic‑reward variants across extensive hyper‑parameter sweeps.

Safety‑zone identification : examined small‑scale test‑time training where intrinsic rewards remain safe.

Model‑collapse‑step metric : extracted a lightweight indicator that measures how many training steps a model survives before collapse.

External‑reward exploration : investigated asymmetric generation‑verification rewards that anchor learning to objective feedback.

Unified Theoretical Framework

All intrinsic‑reward methods share a mechanism that sharpens the model’s initial probability distribution, effectively amplifying existing preferences rather than creating new knowledge. The analysis defines a “confidence‑correctness alignment” (the degree to which the model’s confidence matches correctness). When alignment is high, sharpening improves performance; when low, it accelerates collapse.

Empirical Findings

Across 11 models × 5 intrinsic‑reward variants × multiple hyper‑parameters, a universal “rise‑and‑fall” pattern was observed: early training yields rapid performance gains, followed by an irreversible decline after a critical point. Even the most stable configurations collapse after only a few epochs, indicating a mathematical rather than engineering limitation.

Safety in Small‑Scale Test‑Time Training

In test‑time training with limited data, intrinsic rewards can improve performance without collapse. An extreme experiment used 32 deliberately wrong samples as the training set; despite the proxy reward being wrong from the start, out‑of‑distribution performance still improved steadily, showing that self‑confidence reinforcement stays localized when data are scarce.

Model‑Collapse‑Step Metric

The “model collapse step” counts training steps before the model’s performance collapses under intrinsic rewards. Later collapse correlates with stronger initial priors and higher supervised RLVR gains. This metric predicts a model’s RL‑trainability better than traditional pass@k estimates and requires no ground‑truth labels.

External Reward Alternatives

Two categories of external rewards were explored:

Leveraging massive unlabeled corpora to extract richer reward signals that do not deplete as the model improves.

Asymmetric generation‑verification, where the model’s output is checked by external tools (e.g., compilers, provers, simulators). Preliminary self‑verification experiments showed continuous improvement without the collapse observed for intrinsic rewards.

Key Findings

Confidence‑Correctness Alignment —If the model’s initial bias is correct, sharpening improves performance; if wrong, it accelerates collapse. Collapse is inevitable; only the timing varies.

Small‑Scale Scenarios Are Safer —Intrinsic rewards can stably improve even when the model starts fully incorrect, as demonstrated with 32 wrong samples.

Model‑Collapse‑Step Predicts RL‑Trainability —Later collapse indicates higher suitability for RL; the metric outperforms pass@k without any labeled data.

External Rewards Scale —Verification‑based external rewards break the intrinsic‑reward ceiling, providing objective feedback that continues to guide improvement.

References

Paper: https://arxiv.org/abs/2603.08660

GitHub repository: https://github.com/PRIME-RL/TTRL/tree/urlvr-dev