Tagged articles

Prefix Failure

3 articles · Page 1 of 1

Machine Learning Algorithms & Natural Language Processing

Oct 10, 2026 · Artificial Intelligence

Relay-OPD: Teacher Steps In at Critical Moments to Fix Student Reasoning Errors in Online Distillation

Zhejiang University and Alibaba propose Relay-OPD, an online distillation method that detects student reasoning errors via teacher-student token divergence and briefly hands control to the teacher for targeted correction, achieving +5.73% accuracy on math benchmarks while halving training trajectory length.

Large Language ModelsMathematical ReasoningOn-Policy Distillation

0 likes · 17 min read

Relay-OPD: Teacher Steps In at Critical Moments to Fix Student Reasoning Errors in Online Distillation

Machine Learning Algorithms & Natural Language Processing

Aug 25, 2026 · Artificial Intelligence

When Online Distillation Goes Off‑Track: Relay‑OPD Lets the Teacher Take the Baton

The paper identifies the “prefix failure” problem in on‑policy distillation, proposes Relay‑OPD—a handoff‑triggered, budget‑controlled teacher takeover mechanism that corrects early mistakes, and demonstrates across eight math‑reasoning benchmarks that it improves accuracy by up to 7.3% while halving training trajectory length.

Large Language ModelsOn-Policy DistillationPrefix Failure

0 likes · 13 min read

When Online Distillation Goes Off‑Track: Relay‑OPD Lets the Teacher Take the Baton

Machine Learning Algorithms & Natural Language Processing

Aug 24, 2026 · Artificial Intelligence

When Online Distillation Goes Off‑Track: How Relay‑OPD Lets the Teacher Take Over at Critical Moments

The article analyzes the prefix‑failure problem in on‑policy distillation, introduces Relay‑OPD with a handoff trigger that lets a teacher model intervene locally, and shows through eight math‑reasoning benchmarks that this approach improves accuracy by up to 7.3% while cutting training trajectory length by more than half.

Large Language ModelsOn-Policy DistillationPrefix Failure

0 likes · 13 min read

When Online Distillation Goes Off‑Track: How Relay‑OPD Lets the Teacher Take Over at Critical Moments