Can Your Self‑Distillation Model Do Without Reference Solutions? Introducing d‑OPSD for Diffusion LLMs

Problem with Existing OPSD for Diffusion LLMs

Current on‑policy self‑distillation (OPSD) methods for autoregressive language models inject a reference solution into the teacher’s prompt. This reference acts as privileged information but can cause the student model to hallucinate and to depend on the reference during inference, limiting its ability to generate correct answers.

d‑OPSD: On‑policy Self‑distillation for Diffusion LLMs

d‑OPSD is the first OPSD paradigm designed for diffusion large language models (dLLMs). It eliminates the need for any reference solution or an external teacher model and achieves comparable or superior post‑training performance with only one‑tenth of the reinforcement‑learning (RL) training steps.

The method proceeds as follows:

The student model samples its own future outputs online using the iterative decoding capability of dLLMs.

A random subset of these sampled “self‑future” steps is retained.

The retained steps are fed back to the teacher model, which is constructed from the same student model, providing privileged information that is fully on‑policy.

Supervision is applied at the step level rather than the token level, exploiting the arbitrary‑order generation property of dLLMs and delivering a dense learning signal.

Teacher‑Student Gap Analysis

The relationship between teacher and student is quantified with the TopK‑distribution‑overlap metric. Traditional reference‑solution injection yields a high overlap, restricting the teacher’s ability to introduce new knowledge. d‑OPSD produces a balanced gap: the teacher possesses novel “thinking modes” while maintaining sufficient common language with the student, which facilitates effective knowledge transfer.

Experimental Evaluation

Four mathematical reasoning benchmarks were used to compare d‑OPSD against RL, supervised fine‑tuning (SFT), and other baselines.

Reasoning accuracy: d‑OPSD consistently outperforms the baselines on most tasks.

Training efficiency: RL requires thousands of post‑training steps to converge, whereas d‑OPSD converges within a few hundred steps.

These results demonstrate that d‑OPSD achieves higher performance with substantially lower computational cost.

Resources

Paper: https://arxiv.org/pdf/2606.18195

Code: https://github.com/xingzhejun/d-OPSD