Why Online RL (OPD, RLHF) Uses Reverse KL While Offline SFT Distillation Prefers Forward KL

KL divergence directions

Forward KL (KL(P_teacher || P_student)) weights the expectation by the teacher distribution, giving every high‑probability teacher token a chance to appear in the training data and provide gradient signals. This pushes the student to cover all major teacher modes.

Reverse KL (KL(P_student || P_teacher)) weights the expectation by the student distribution; training samples from the student and then measures divergence to the teacher. The sampled subset concentrates on a few dominant modes (mode‑seeking). Tokens that the teacher assigns high probability but the student samples rarely receive little or no gradient information.

“Multimodal” refers to multiple peaks in a probability distribution, not image‑text multimodality.

SFT / pre‑training distillation

The goal is for the student to learn the teacher’s full capability for downstream tasks or to produce a cheaper “degraded” teacher. Consequently forward KL is chosen so the student attempts to cover every teacher mode.

Online Policy Distillation (OPD) and RLHF

Both industry and academia favor mode‑seeking. In industry, OPD merges several teachers (e.g., GLM, DS, Kimi) whose policies are already concentrated on specialized abilities. The resulting multimodal target consists of sharp, specialist distributions. Covering all modes is infeasible, so reverse KL is used to sacrifice less important modes and focus on dominant ones, guided by data filtering or teacher weighting.

Academically, OPD often aims for a small student to match or surpass a large teacher on selected tasks. Limited capacity forces the student to discard low‑priority modes and concentrate capacity on core tasks, again using reverse KL.

Common misunderstanding

Because the teacher’s probabilities are normalized, one might expect that tokens never sampled by the student would still receive gradients via the softmax and eventually acquire the teacher’s ability. In practice, when the student’s high‑probability modes are suppressed, probability mass shifts to nearby secondary modes rather than to unseen ones, leaving large‑probability teacher tokens inaccessible to gradient updates.

Top‑k and full‑vocabulary OPD

Recent work on top‑k and full‑vocabulary OPD mitigates some of these issues, but it cannot fully solve them because language models define a joint probability over entire token sequences; covering a single token position does not guarantee coverage of the whole sequence.

Reference

[1] https://tminka.github.io/papers/message-passing/minka-divergence.pdf