Breaking OPD’s Teacher Ceiling with MAD‑OPD: Small Models Learn Debated Answers

Background and limitation of OPD

On‑Policy Distillation (OPD) trains a student model by having it generate actions on its own trajectory and then receiving token‑wise supervision from a teacher conditioned on the student’s prefix. This adapts to the student’s parameter distribution, but when supervision comes from a single teacher the student is limited by any blind spot of that teacher. Errors in the teacher (e.g., wrong dates in tool calls, inconsistent branches in code generation) become training targets for the student. In agentic scenarios a single erroneous tool call can corrupt the entire multi‑turn dialogue and downstream trajectory.

MAD‑OPD design

To break the single‑teacher ceiling, MAD‑OPD introduces multiple teachers that debate around the student’s current state. The debate proceeds as follows:

The student samples an action (or a full code generation) at the current state.

The environment returns an observation (OPAD).

Each teacher generates a criticism, corrects the others, and produces a confidence score.

The teachers aggregate their outputs into a consensus distribution weighted by the confidence scores.

The consensus is force‑decoded as the target for the student; the debate transcript is kept only for the teachers.

For agentic tasks the debate‑augmented OPD (OPAD) is applied at every step, handling long trajectories. For code generation the reverse KL divergence is used to concentrate the student on a single implementation path, while for agentic tasks Jensen‑Shannon Divergence (JSD) is chosen to avoid gradient explosion on divergent states.

Experimental results

Evaluations on agentic benchmarks (BFCL‑v4, τ²‑Bench, VitaBench) and code benchmarks (LiveCodeBench v6, MBPP+) show that MAD‑OPD consistently outperforms the base OPD and the multi‑teacher OPD (MT‑OPD). On a Qwen‑3 14B+8B → 4B student the scores are:

Agentic Avg: 25.69 (MAD‑OPD) vs 23.26 (OPD) vs 24.14 (MT‑OPD) vs 18.65 (Base)

Code Avg: 48.12 (MAD‑OPD) vs 44.41 (OPD) vs 41.97 (MT‑OPD) vs 41.50 (Base)

Overall Avg: 34.66 (MAD‑OPD) vs 31.72 (OPD) vs 31.27 (MT‑OPD) vs 27.79 (Base)

On LiveCodeBench v6, the 4B student trained with MAD‑OPD reaches pass@1 = 29.83, surpassing the 14B teacher’s 25.57 by 4.26 points; under BoN@16, MAD‑OPD 4B scores 43.43 versus 33.14 for the teacher.

Ablation study

Removing the debate step reduces performance, confirming that the gain is not merely from adding more teachers. MT‑OPD’s Code Avg (41.97) is lower than single‑teacher OPD (44.41) because naïve averaging harms code consistency. Adding debate raises Code Avg to 46.60; incorporating confidence weighting further improves it to 48.12. Two rounds of debate give the best trade‑off; three rounds degrade performance due to context bloat and noise.

Additional ablations on divergence choice show that JSD stabilizes training on long agentic trajectories, acting as a “shock absorber”, whereas reverse KL is preferable for code generation where a single coherent path is needed.

Conclusion

MAD‑OPD upgrades the supervision source from a single‑teacher conditional distribution to a consensus distribution obtained after multi‑teacher debate. Teachers become mutual error detectors, allowing small models to inherit complementary correctness from a pool of teachers and break the previous performance ceiling.