When Online Distillation Goes Off‑Track: Relay‑OPD Lets the Teacher Take the Baton

As large language models become stronger, efficiently transferring their capabilities to smaller models in the post‑training stage is a key challenge. On‑policy distillation (OPD) provides dense token‑wise supervision on the student’s own trajectory, mitigating distribution shift between training and inference. However, the authors observe a critical issue: once a student makes a wrong decision early in a long‑chain reasoning task, all subsequent tokens are generated from that erroneous prefix, causing the teacher’s supervision to become increasingly unreliable and wasting computation on off‑track continuations.

Existing remedies—fixed‑length truncation (ESR, FastOPD), offline rewrite (TRD), and token‑level mixing (SKD)—either cut at irrelevant positions, intervene too late, or lack an explicit signal that the reasoning direction has already failed. The authors note a striking divergence: when a prefix is wrong, teachers tend to emit reflective tokens such as “But”, “However”, or “Wait”, while students continue with the original (incorrect) direction. In a typical case the teacher would choose a reflective token with 74.4% probability versus the student’s 50.6% for a continuation token.

To exploit this signal, the team introduces Relay‑OPD (Relay On‑Policy Distillation). It consists of three key designs:

Label‑free handoff trigger. A predefined list R of reflective words (e.g., Wait, But, However) is used. When the teacher’s top‑1 next‑token belongs to R while the student’s top‑K (K=5) does not, a handoff is triggered.

Relay budget (M, L). At most M teacher takeovers are allowed per rollout. Each takeover starts with the triggered reflective token and generates L natural paragraphs (≈23 tokens each), ensuring each segment ends at a structurally complete reasoning unit. The main experiments use (M, L) = (2, 3).

Training objective on the relay trajectory. Distillation is performed only on the actual student‑generated tokens that are corrected by the teacher, rather than fitting the teacher’s full output distribution, because the teacher’s guidance is unreliable on the student’s own prefix.

Implementation leverages a speculative decoding engine: the student acts as the draft model and the teacher as the target model, allowing the teacher’s logits to be consulted at every student token without extra synchronization overhead. This yields a single‑engine rollout whose distribution matches that of a true alternating generation.

Experiments use Qwen3‑4B‑Instruct‑2507 as the teacher and Qwen3‑0.6B / Qwen3‑1.7B (non‑thinking) as students, trained on the DAPO‑Math‑17K dataset with verl and vLLM. Evaluation spans eight math‑reasoning benchmarks (AIME 2024‑2026, MATH500, AMC 2023, OlympiadBench, HMMT 2026) without any verifier or correctness labels. Relay‑OPD achieves the best or second‑best score on all benchmarks, improving average accuracy from 41.23% (standard OPD) to 46.96% (+5.73%) for the 1.7B student. Notable gains include +7.29% on AIME 2025 and +7.19% on AIME 2026, surpassing the strongest baseline FastOPD (+1.49%). Training trajectory length is reduced by >50% (2,296 tokens vs. 4,658 for OPD) and the method reaches the optimal checkpoint in 35 steps instead of 55.

Further analysis shows:

Local correction at each handoff (teacher token accounts for only 0.35% of total tokens) raises accuracy from 27.73% to 34.96% (+7.23%).

Delaying the handoff reduces accuracy dramatically (41.99 → 33.98 → 29.49), confirming that early intervention is essential.

Increasing L beyond 4 degrades performance, while M = 2 is optimal; larger M harms the on‑policy advantage.

Efficiency-wise, teacher token proportion drops from ~13% initially to 2‑3% after ~20 steps, indicating that as the student improves it naturally requires fewer teacher takeovers. The strategy entropy remains higher than OPD and FastOPD, reflecting richer exploration induced by teacher corrections.

All code, including training, ablation, and evaluation scripts, is open‑sourced (https://github.com/ZJU-REAL/Relay-OPD) and the paper (https://arxiv.org/abs/2607.26057) provides full details. Subsequent work on agent distillation has already adopted and extended the handoff mechanism (https://arxiv.org/abs/2608.01953).