From Imitation to Optimization: Recent Advances in On-Policy Distillation

Introduction

When reinforcement‑learning exploration becomes costly, On‑Policy Distillation (OPD) lets a student model learn from its own sampled trajectories while a teacher provides dense token‑level supervision, avoiding distribution mismatch and replacing sparse sequence‑level rewards with immediate per‑token signals. However, reverse KL’s mode‑seeking bias can reduce diversity, advantage‑weighted gradients may vanish near zero advantage, and mismatched teacher‑student reasoning can cause negative transfer.

1. Training Stabilization

Entropy‑Aware On‑Policy Distillation (EOPD)

Paper: https://arxiv.org/abs/2603.07079. EOPD observes that high‑entropy teacher distributions encode valuable uncertainty, especially for mathematical reasoning where multiple continuations are plausible. It applies reverse KL on low‑entropy tokens for precise imitation and switches to forward KL on high‑entropy tokens to encourage broader exploration. Experiments on several math‑reasoning benchmarks show that EOPD preserves generation entropy and improves Pass@8.

Asymmetric On‑Policy Distillation (AOPD)

Paper: https://arxiv.org/abs/2605.06387. AOPD identifies three structural issues with advantage‑weighted policy‑gradient updates: high variance, rapid decay of gradients when advantage≈0, and reliance on negative‑advantage updates when teacher corrections are scarce. It keeps policy‑gradient updates for tokens with positive advantage and replaces negative‑advantage updates with a local KL minimization that directly aligns the student to the teacher. This asymmetric framework yields average gains of 4.09 pp (strong initialization) and 8.34 pp (weak initialization) on math tasks while maintaining higher policy entropy.

Relaxed On‑Policy Distillation (REOPOLD)

Paper: https://arxiv.org/abs/2603.11137. REOPOLD shows that stop‑gradient OPD is equivalent to a policy‑optimization form where the teacher‑student log‑likelihood ratio acts as a token‑level reward. It introduces three designs: (1) Mixture‑based Reward Clipping to curb extreme negative feedback, (2) Entropy‑guided Dynamic Sampling to focus on high‑uncertainty tokens, and (3) a two‑stage Exploration‑to‑Refinement schedule that weakens negative rewards early and sharpens supervision later. REOPOLD achieves 6.7–12× sample‑efficiency over recent RL baselines, lets a 7B student approach a 32B teacher, and delivers >3× inference speed‑up.

2. Self‑Distillation

Unified Self‑Distillation (UniSD)

Paper: https://arxiv.org/abs/2605.06597. UniSD builds a systematic analysis framework that integrates five mechanisms: Multi‑Teacher Agreement, EMA Teacher Stabilization, Token‑level Contrastive Learning, Feature Matching, and Divergence Clipping. Ablations reveal EMA Teacher as the most impactful component. The full configuration (UniSD_full) improves average performance by 5.4 pp over the baseline and by 2.8 pp over the strongest prior self‑distillation method across six benchmarks and three model families.

On‑Policy Self‑Distillation (OPSD)

Paper: https://arxiv.org/abs/2601.18734. OPSD lets a single LLM act as both teacher and student. The student samples its own answers; the teacher receives privileged trajectories (e.g., reference solutions) and provides a per‑token distribution. The student’s distribution is aligned to the teacher’s via per‑token KL minimization. This eliminates the need for an external teacher, mitigates exposure bias from supervised fine‑tuning, and replaces sparse RL rewards with dense token‑level signals. Experiments report an 8–12× token‑efficiency gain over GRPO and offline distillation methods on multiple math reasoning benchmarks.

3. Mechanism Analysis

The Many Faces of OPD: Pitfalls, Mechanisms, and Fixes

Paper: https://arxiv.org/abs/2605.11182. Three failure mechanisms are identified: (1) distribution mismatch because the teacher conditions on student‑generated prefixes, (2) instability from biased Top‑K reverse‑KL gradients, and (3) OPSD’s limitation that the student learns a “no‑PI” policy that does not generalize when PI is instance‑specific. Remedies include stop‑gradient Top‑K objectives, RL‑adapted teachers, and SFT‑stable students, which together alleviate gradient bias and prevent collapse.

Rethinking OPD: Phenomenology, Mechanism, and Recipe

Paper: https://arxiv.org/abs/2604.13016. Successful OPD requires (i) compatible teacher‑student reasoning modes and (ii) the teacher to provide capabilities absent from the student’s prior training. Overlap analysis shows that in successful runs the high‑probability token overlap rises from ~72 % to ~91 % and the entropy gap shrinks, whereas failed runs stall early. Recovery strategies are (i) off‑policy cold‑start with teacher roll‑outs to boost initial overlap, and (ii) prompt selection that aligns teacher and student distributions (with a trade‑off of reduced entropy).

Token Importance in OPD (TIP)

Paper: https://arxiv.org/abs/2604.14084. TIP argues that the most informative tokens belong to two regions: (1) high‑entropy student positions (uncertainty) and (2) low‑entropy positions with large teacher‑student divergence (over‑confident mistakes). A soft‑OR scoring that combines entropy and KL divergence selects a small subset of tokens that retains or exceeds full‑token performance. Using only 50 % of tokens matches full training, while <10 % of “low‑entropy, high‑divergence” tokens achieve near‑baseline results. Validation spans Qwen3, Llama, Qwen2.5, MATH‑500, AIME 2024/2025, and DeepPlanning, with 20 % of tokens sometimes surpassing full‑token OPD.

Conclusion

Recasting OPD as a policy‑optimization problem unifies entropy‑aware, asymmetric, and token‑selection ideas, enabling stable training while preserving diversity and exploration. Future progress may depend on accurately identifying valuable learning signals, balancing precise imitation with free exploration, and enabling models to self‑surpass when stronger teachers are unavailable.