Why LLM Post-Training Got Hard: 5 Paradigm Shifts in 6 Months

The past six months have seen more change in large-model post-training than the previous three years combined. The author reviewed nearly every recent paper in the field and identifies five paradigm-level shifts that redefine how models are fine-tuned.

Change 1: From One Model to a Crowd of Experts

Traditional post-training packs math, code, and agent RL into a single run. These capabilities conflict: math demands rigorous convergence, code needs exploration, agent tasks require long-horizon planning — resulting in a see-saw effect and prohibitive cost.

The 2026 solution is counter-intuitive: train domain experts separately, then distill them back into a generalist. Xiaomi's MiMo-V2-Flash technical report introduces MOPD (Multi-Teacher On-Policy Distillation): domain experts act as teachers; the student samples its own trajectories and is optimized with token-level reverse KL plus outcome rewards.

Change 2: Online Distillation as RL's "Cheap Substitute"

RL suffers from sparse rewards — a model generates 2,000 tokens and receives only a final correct/incorrect signal. Online distillation provides dense feedback: the teacher evaluates every token of the student's trajectory.

One RL run costs ~17,920 GPU hours; online distillation needs ~1,800 GPU hours — roughly a 10× reduction.

Change 3: RLVR Enters a "Refinement" Phase

If 2025 RLVR was about stacking compute, 2026 H1 is about training smarter. Six concrete directions:

① Entropy Collapse — STEER

Models become homogeneous, losing diversity. Naively raising entropy causes hallucination. STEER derives a token-level entropy-change analytic form and only re-weights tokens with sharp entropy drops. On Qwen2.5-Math-7B, average score rises from 44.2 to 48.6. Awarded ACL 2026 Outstanding Paper.

② Insufficient Exploration — I²B-LPO

Sampling 16 trajectories per prompt yields near-identical paths. I²B-LPO locates "high-entropy hesitation nodes" (where the model truly hesitates), injects latent variables to force branching, and uses an information bottleneck to reward valid trajectories. Accuracy +5.3%, diversity +7.4%.

③ Problem Selection — GPS

Rollouts are expensive; most problems are too easy (no learning signal) or too hard (unlearnable). GPS trains a tiny 20M model that shares "difficulty knowledge" across prompts, selecting medium-difficulty diverse problems. Cuts rollouts by up to 69%.

④ Capability Boundary Collapse — Boundary-Aware Curriculum RL

Pass@1 rises on training data while pass@256 falls below the base model — the model merely amplifies already-known solutions and forgets hard problems. This method uses pass@256 to locate the true capability boundary and guides training on boundary problems. Pass@256 improves by +9.8 percentage points on average.

⑤ Catastrophic Forgetting — RECAP

Reasoning improves but writing, translation, and chat degrade. RECAP replays general data and dynamically re-weights by "target convergence" and "instability." General ability is preserved while reasoning further improves.

⑥ Parameter Efficiency — GeoRA

RLVR updates are sparse and systematically avoid pre-training's principal directions. GeoRA uses geometric priors to locate this subspace, then compresses it via SVD into a low-rank dense adapter. 0.5% of parameters achieve near full-parameter performance. Also an ACL 2026 Outstanding Paper.

Change 4: SFT Is No Longer a Harmless Warm-up

Old assumption: SFT just teaches the model to follow instructions; RL does the real work. PEAR disproves this: a stronger SFT checkpoint can yield worse final RL performance. Cause: distribution mismatch — SFT trains on offline data, RL samples online; the deeper SFT learns, the further it drifts from the RL distribution. Fix: importance-sampling re-weighting of SFT loss. AIME-2025 pass@8 gains +14.6%.

BRIDGE goes further with bi-level optimization: SFT learns to "supervise RL" — passing only knowledge useful for reward optimization. On Qwen2.5-3B, training speeds up 44% and performance rises 13%.

Change 5: Data Quality Is a Hard Constraint

Prior work claimed RLVR is robust even with 100% noisy labels. A 2026 re-audit of those datasets found at least 16% clean data hidden in the "pure noise" sets. Training on truly wrong labels drops math benchmarks 8–10% and Text2SQL 5–12%. SOTA algorithms (Dr.GRPO, DAPO, TIS, SAPO, PGFC) cannot recover the loss. Conclusion: no algorithmic cleverness compensates for data quality; high-quality data remains a hard constraint.

The field has shifted from "who has more GPUs wins" to "who understands training dynamics wins." Entropy dynamics, capability boundaries, distribution alignment, and data cleanliness — once ignored details — are now the decisive levers. For practitioners, this is good news: the leverage of understanding has, for the first time, exceeded the leverage of raw compute.