How DOPD Overcomes the Privilege Illusion to Boost Online Policy Distillation

Online policy distillation has become popular because dense token‑level teacher signals can transfer ability to a student. However, injecting privileged information (e.g., hints, visual annotations) can create a “privilege illusion”: the teacher appears stronger only due to information asymmetry, leading to entropy collapse, performance degradation, and poor generalization.

The paper “DOPD: Dual On‑policy Distillation” (arXiv:2606.30626) from NUS, CUHK‑MMLab, Peking University and JD Exploration proposes a dual‑distillation paradigm that perceives privilege advantage. DOPD computes a Privilege Advantage Gap between teacher and student forward passes and combines it with teacher‑student confidence to dynamically select one of four token‑wise distillation strategies.

Token‑wise strategies :

High advantage + teacher high confidence (key knowledge tokens) : teacher‑centric distillation with full‑vocab Jensen‑Shannon divergence.

Low advantage + both high confidence (consensus tokens) : lightweight teacher distillation with Top‑K reverse KL.

Low advantage + both low confidence (uncertain tokens) : weak self‑distillation with Top‑K reverse KL to avoid drift.

High advantage + student high confidence (exploratory tokens) : lightweight self‑distillation with Top‑K reverse KL to preserve student exploration.

Experiments on eight LLM benchmarks show DOPD improves the student average score by 7.5 points, closing 89.8 % of the initial teacher‑student gap, and outperforms nine SOTA OPD variants. On VLM benchmarks, DOPD gains 6.0 points and recovers 69.2 % of the gap. When the size disparity is large (8 B → 0.6 B), the gain is 14.1 points—four times the improvement of vanilla OPD.

Additional studies demonstrate that DOPD yields stable training curves, fast convergence (reaching 200‑step performance by step 80), and healthy entropy dynamics (initial rise for exploration, then gradual decline). Sensitivity analysis shows performance is robust to a wide range of hyper‑parameters, and ablation experiments confirm each component contributes to the final gain.

Analysis of privileged information types reveals that providing only the final answer harms learning, while “ability‑oriented” hints that guide without revealing answers achieve the best results, further validating the privilege illusion hypothesis.

Visualizations of token classification align with intuitive expectations, confirming that DOPD’s token groups have semantic meaning. The paper also reports that DOPD maintains superior continual‑learning performance across three distillation stages and achieves 3–4 points higher out‑of‑distribution generalization than the second‑best method.

Overall, DOPD rethinks online policy distillation by introducing advantage‑aware dual distillation, offering a scalable and robust solution for LLM and VLM student models.