When Online Distillation Goes Off‑Track: How Relay‑OPD Lets the Teacher Take Over at Critical Moments

As large language models become stronger, transferring their capabilities to smaller models efficiently after training is a key challenge. On‑policy distillation (OPD) provides dense token‑wise supervision on the student’s own generated trajectories, but the student’s early mistakes can cause the entire rollout to drift, making the teacher’s supervision unreliable and wasting compute.

Existing remedies—fixed‑length truncation (ESR, FastOPD), offline rewrite (TRD), and token‑level mixing (SKD)—either cut at the wrong positions, intervene too late, or lack an explicit signal that the reasoning prefix has already failed.

Observing that, when a prefix is off‑track, teachers tend to pause and emit reflective tokens such as “Wait”, “But”, or “However”, while students continue the erroneous direction, the authors define a handoff trigger : if the teacher’s most probable next token belongs to a predefined reflection set R and the student’s top‑K candidates do not contain any token from R (K=5 in the main experiments), a handoff is triggered.

Relay‑OPD builds on this trigger with three design components:

Label‑free handoff trigger : a reflection word list R (e.g., Wait, But, However) determines when the teacher should intervene.

Handoff budget (M, L) : at most M handoffs are allowed; each handoff starts with the triggered reflective token and generates L natural paragraphs (average 23.2 tokens per paragraph). The main setting uses (M, L) = (2, 3).

Training objective on handoff trajectories : the student distills only from the teacher’s corrections on the actual handoff segments, rather than trying to match the teacher’s full output distribution.

Implementation merges the draft (student) and target (teacher) models into a single speculative decoding engine, eliminating the need for a separate verifier or extra communication overhead.

Experiments use Qwen3‑4B‑Instruct‑2507 as the teacher and Qwen3‑0.6B / Qwen3‑1.7B as students, evaluated on eight English math‑reasoning benchmarks (AIME 2024‑2026, MATH500, AMC 2023, OlympiadBench, HMMT 2026). No verifier, process‑label, or answer‑correctness label is used. Relay‑OPD achieves the best or second‑best score on all benchmarks, with an average accuracy of 46.96 % for the 1.7B student (+5.73 % over standard OPD) and reduces average training trajectory length from 4,658 tokens to 2,296 tokens (‑50.7 %).

Compared with baselines, Relay‑OPD outperforms FastOPD (+1.49 % on average), TRD (significantly weaker due to visible rewrite artifacts), and SKD (fails to break the student’s repetitive generation pattern). It also shortens inference length (‑17.9 % to ‑28.3 % on AIME 2025/2026 and HMMT) while improving accuracy (+2.39 % to +4.17 %).

Ablation studies show that local correction is crucial: adding a single teacher token at each trigger raises accuracy from 43.48 % to 46.25 % (+2.77 %). Both the handoff segment and the handoff‑specific training objective are indispensable; removing either degrades performance. Sensitivity analysis reveals optimal values at M = 2 and L = 4; larger values cause over‑correction and diminish the on‑policy advantage.

Further observations include a self‑adaptive decay of teacher intervention intensity (teacher token proportion drops from ~13 % early to 2‑3 % after ~20 steps) and higher policy entropy for Relay‑OPD, indicating richer exploration.

All code, models, and evaluation scripts are open‑source (https://github.com/ZJU-REAL/Relay-OPD), and the handoff signal is obtained directly from teacher‑student logit comparisons without any external verifier.