When Online Distillation Goes Off‑Track: Relay‑OPD Lets the Teacher Take the Baton
The paper identifies the “prefix failure” problem in on‑policy distillation, proposes Relay‑OPD—a handoff‑triggered, budget‑controlled teacher takeover mechanism that corrects early mistakes, and demonstrates across eight math‑reasoning benchmarks that it improves accuracy by up to 7.3% while halving training trajectory length.
