Road MOPD: Harmonizing Conflicting Expert Teachers in Multi-Domain Distillation

Problem: Teacher Diversity Causes Gradient Conflicts in Multi-Teacher Distillation

In multi-teacher online policy distillation (MOPD), domain-specific expert teachers (e.g., math, code, medical) supervise a single student model. While domain routing separates supervision sources, their updates converge on the same shared parameters. Teachers differ in reasoning style, response format, and token-level preferences, causing gradient conflicts that degrade performance in one domain while another improves. The article shows that for math+code the average gradient-conflict layer ratio is only 2.2%, and joint training mostly matches or exceeds single-domain baselines. However, for math+medical and code+medical the conflict ratios jump to 45.2% and 36.6% respectively, and all four medical benchmarks fall below single-domain online policy distillation (OPD) scores (Figure 1). This demonstrates that benefits observed in compatible domain pairs do not automatically transfer to high-conflict pairs.

Road MOPD: Cross-Teacher Coordination Across Supervision and Update

Road MOPD retains each domain expert as the primary supervisor for its own domain, but adds auxiliary teachers that evaluate the same student response. The method operates in two stages (Figure 2):

Step 1: Token-Level Consensus Adjustment

For each token, the method compares the primary teacher's and auxiliary teacher's probability shifts relative to a frozen reference model (the student at initialization). If both teachers increase (or both decrease) the token's probability, the direction is consistent; if one increases while the other decreases, they disagree. This agreement signal is smoothed, clipped, and turned into a modulation coefficient that scales the primary teacher's relative reward. Consensus positions are amplified; divergent positions are attenuated. When evidence is near zero, the modulation reverts to standard OPD. This captures cases where absolute probabilities differ but correction directions align (Figure 3). Empirically, batching samples by teacher disagreement shows medical gradient-conflict rates rising from 46% to 76%, confirming that textual disagreement predicts update conflict.

Step 2: Preventive Gradient Projection with Norm Floor

After token-level modulation, the adjusted supervision yields a target gradient. The auxiliary teacher's standard OPD gradient on the same response provides an auxiliary gradient. For each parameter tensor, the method checks the inner product: if compatible, the target gradient is kept; if conflicting, the conflicting component is removed. Crucially, the auxiliary gradient only constrains direction—it is not added as an independent update. This differs from PCGrad, which projects gradients from different tasks on their own data; Road MOPD projects based on other teachers' feedback on the current response (Figure 4). To avoid over-suppressing the target domain's learning signal, a norm-preservation floor is set: the projected gradient's norm cannot fall below a fraction of the original target gradient's norm. Ablation shows that removing this floor (full projection) drops composite scores by 0.83 and 0.44 points in the two high-conflict settings. With multiple auxiliary teachers, consistencies are aggregated and projections applied sequentially within the same norm floor.

Experimental Setup

Student models: Qwen3-4B and Qwen3-1.7B (thinking mode disabled).

Domains & benchmarks: Math, Code, Medical — 11 benchmarks total.

Baselines: Vanilla MOPD, ExOPD, Open-MOPD, CaMOPD.

Evaluation: Domain averages computed per benchmark; composite score is equal-weight average of domain averages. Medical results from text QA benchmarks.

Results on High-Conflict Pairs (Qwen3-4B)

Math + Medical: vs. Vanilla MOPD, math average rises from 45.97 to 47.60 (+1.63 pp), medical from 39.45 to 48.23 (+8.78 pp). Composite from 42.71 to 47.92 (Figure 5).

Code + Medical: code average from 60.53 to 63.02 (+2.49 pp), medical from 40.75 to 47.75 (+7.00 pp). Composite from 50.64 to 55.38. In both pairs, Road MOPD leads all compared multi-domain methods on both domain averages.

Results on Smaller Model (Qwen3-1.7B)

Code+Medical still shows mutual gains: code +2.15 pp, medical +5.58 pp. Math+Medical exhibits a trade-off: math drops from 28.36 to 27.21, medical rises from 31.18 to 36.89, but composite improves from 29.77 to 32.05. Across four high-conflict experiments (two model sizes × two pairs), Road MOPD achieves the highest composite score, leading the strongest baseline by 0.72–2.89 points. Math and code averages on 1.7B remain slightly below ExOPD, but equal-weight composite is better.

Extension to Low-Conflict and Three-Domain Settings

Math + Code (low conflict): On 4B, math +1.61 pp, code +1.83 pp, composite reaches 55.40, showing gains extend to compatible pairs.

Math + Code + Medical (three domains): On 4B, domain averages vs. Vanilla MOPD improve by +0.31 (math), +1.68 (code), +7.42 (medical). Composite 52.89 vs. Vanilla 49.76 (+3.13) and vs. second-best CaMOPD 51.71 (+1.18). All three domain averages lead all compared methods (Figure 6).

Conclusion

Road MOPD addresses the "learn one domain, forget another" problem by threading cross-teacher feedback through both supervision (token-level consensus modulation) and optimization (preventive gradient projection with norm floor). The approach lets a single student absorb diverse expert capabilities while minimizing mutual interference. Results on Qwen3 models demonstrate consistent composite-score gains across high-conflict, low-conflict, and three-domain combinations, with medical improvements up to 8.78 percentage points.

References & Resources

Paper: https://arxiv.org/abs/2610.13521 Code: https://github.com/jokersio-tsy/RoadMOPD Hugging Face:

https://huggingface.co/papers/2610.13521

}