LLM Post-Training Paradigm Shift: 5 New Paths Replacing Monolithic RL

Over the past half year, large-model post-training has undergone a paradigm-level change. The author reviewed nearly all recent post-training papers and identifies five major paths.

Change 01: From One Model to a Group of Experts

Traditionally, post-training meant taking a single base model and training math RL, code RL, and agent RL together in one run. These capabilities conflict: math demands rigorous convergence, code needs exploration, agent tasks require long-horizon planning — a classic seesaw where improving one degrades another, at exorbitant cost.

The 2026 solution is counter-intuitive: train them separately. Xiaomi's MiMo-V2-Flash technical report introduces MOPD (Multi-Teacher On-Policy Distillation). Domain experts act as teachers; the student model samples its own trajectories and is optimized with token-level reverse KL plus outcome rewards.

Change 02: Online Distillation Becomes RL's "Cheap Alternative"

RL suffers from sparse rewards — a model generates 2,000 tokens and receives only a final correct/incorrect signal, with no feedback on intermediate reasoning steps. Online distillation provides dense feedback: the teacher model evaluates every token of the student.

One RL run costs ~17,920 GPU-hours; online distillation needs only ~1,800 GPU-hours — roughly a 10x reduction.

Change 03: RLVR Enters a "Refinement" Phase

If 2025 RLVR was about "stacking compute," 2026 H1 is about "getting smarter" in six specific directions:

① Entropy collapse — model outputs homogenize. STEER derives a token-level entropy-change analytic formula and applies heavy reweighting only to tokens with sharp entropy drops. On Qwen2.5-Math-7B, average score rises from 44.2 to 48.6; awarded ACL 2026 Outstanding Paper.

② Insufficient exploration — sampled trajectories are nearly identical. I²B-LPO identifies high-entropy "hesitation nodes" in reasoning, injects latent variables to force branching, and uses an information bottleneck to reward diversity and filter invalid trajectories. Accuracy +5.3%, diversity +7.4%.

③ Question selection — which problems deserve compute. GPS trains a tiny 20M model that shares "difficulty knowledge" across prompts, prioritizing medium-difficulty diverse problems. Cuts rollout by up to 69%.

④ Capability boundary collapse — pass@1 rises but pass@256 falls. The model merely increases probability of already-solved problems while losing ability on hard ones. Boundary-Aware Curriculum RL uses pass@256 to locate the true capability boundary and guides training on boundary problems. pass@256 improves by +9.8 percentage points on average.

⑤ Forgetting — reasoning improves but general skills degrade. RECAP replays general data and dynamically reweights by "target convergence" and "instability." General ability is preserved while reasoning further improves.

⑥ Parameter efficiency — 0.5% of parameters achieve near full-parameter performance. GeoRA observes that RLVR updates are sparse and systematically avoid pre-trained principal directions. It uses geometric priors to locate this subspace and compresses it into a low-rank dense adapter via SVD. 0.5% parameters match full-parameter results; also an ACL 2026 Outstanding Paper.

Change 04: SFT Is No Longer a "Harmless Warm-up"

The old assumption: SFT just teaches the model to follow instructions, then RL takes over; SFT quality doesn't matter. This is now disproven.

PEAR proves a counter-intuitive result: a "stronger SFT checkpoint" can perform worse after identical RL. Cause: distribution mismatch — SFT trains on offline data, RL samples online; the deeper SFT learns, the further it drifts from the RL distribution. Fix: importance-sampling reweighting of SFT loss. AIME-2025 pass@8 gains up to +14.6%.

BRIDGE goes further with bi-level optimization, making SFT "learn to supervise RL" — passing only knowledge useful for reward optimization. On Qwen2.5-3B, training is 44% faster with +13% performance.

Change 05: Data Quality Is an Unavoidable Hard Constraint

Previous work claimed RLVR is robust even with 100% noisy labels. A 2026 study rigorously re-audited those datasets and found at least 16% clean data mixed into the "pure noise" sets. Training on truly wrong labels drops math benchmarks by 8–10% and Text2SQL by 5–12%. SOTA algorithm improvements (Dr.GRPO, DAPO, TIS, SAPO, PGFC) cannot recover the loss. Conclusion: no algorithmic cleverness can substitute for high-quality data.

The essence of this half-year shift: the field moves from "who has more GPUs wins" to "who understands training dynamics wins." Entropy dynamics, capability boundaries, distribution alignment, data cleanliness — details once ignored — are now the decisive levers. For practitioners, this is good news: the leverage of understanding has, for the first time, exceeded the leverage of compute.