xOPD Evolution: Mapping Recent OPD Improvements – Rephrased Same Problems vs. New Modules

AOPD: Splitting Advantage by Sign

Li et al. [1] observe that the policy‑gradient signal in GRPO varies dramatically with the sign of the advantage: zero advantage yields vanishing gradients, negative advantage produces high‑variance noise, and only positive advantage provides a reliable learning signal. They split the advantage into two parts—retain the policy‑gradient loss for positive advantage (exploitation) and replace the loss for non‑positive advantage with a direct KL‑minimization (imitation). Experiments report a 4.09 % gain with strong initialization and an 8.34 % gain with weak initialization.

SOD: Decaying Teacher Signal for Tool‑Use Agents

Hy et al. [2] target a failure mode in tool‑call scenarios where a hallucinated tool call corrupts the trajectory, making the teacher’s supervision noisy. They partition the trajectory at each tool observation into K + 1 steps, compute a divergence score for each step, and use the ratio of successive scores as an adaptive weight: the teacher signal is attenuated when the student diverges and restored when alignment improves. This “trust gate” yields a 20.86 % relative improvement over vanilla OPD on a 0.6 B model in the AIME2025 benchmark.

ROPD: Questioning Logit‑Based Teacher Signals

NUS + Tencent propose ROPD [3], which removes the assumption that teacher signals must be logits. For each prompt they collect four teacher answers and eight student answers, let an LLM generate 4‑12 weighted rubric items, and feed the rubric‑derived 0/1 rewards directly to GRPO. The rubric reward achieves an AUC of 0.90 versus 0.35 for teacher logits, delivering a 10× sample‑efficiency boost and a 6.3× wall‑clock speedup.

Apple Unmasking OPD: Diagnosing Gradient Noise

Apple’s unmasking study [4] measures cosine alignment between teacher and student gradients. On failed student trajectories the alignment is ~0.05 (significant positive), while on successful trajectories it drops to ~0.001 (near‑orthogonal), indicating wasted gradient budget. Token‑level variance is high (std ≈ 0.83) and alignment oscillates between positive and negative. Retaining only positively aligned tokens captures 52 % of tokens but yields a 10‑15× effective signal.

Uni‑OPD: Bridging Token‑Level and Trajectory‑Level Distillation

Hy’s Uni‑OPD [5] identifies a hidden bug in vanilla OPD: token‑level reverse KL does not always correlate with final outcomes. They introduce outcome‑guided margin calibration, defining a trajectory‑level reward that must stay consistent with the outcome reward, and adjust gradients until the margin is satisfied. Multi‑teacher experiments (30 B → 4 B/1.7 B) show 1.5‑3 pt gains.

Lightning OPD: Caching Teacher Inference

The Lightning OPD trick [6] observes that the dominant cost of OPD is teacher inference, not KL computation. By caching teacher logits from the SFT stage and reusing them during OPD, they achieve a 4× compute saving and reach 69.9 % on AIME24 with Qwen‑3‑8B using 30 GPU·hr. The method is orthogonal to other xOPD techniques and can be stacked on any approach.

OPSDL: Self‑Distillation for Long‑Context Scenarios

Baidu’s OPSDL [7] addresses the problem that long‑context rollouts are polluted by irrelevant information, while short‑context rollouts remain reliable. Instead of an external long‑context teacher, the model acts as its own teacher: the student generates a full‑context rollout, a short‑context slice is extracted, and the model provides supervision on that slice. This “privileged information” aligns with Apple’s comprehensibility hypothesis that a teacher signal is useful only when the student can parse it.

Self‑Distilled Reasoner: Forward KL Wins in Self‑Distillation

UCLA + Meta’s Self‑Distilled Reasoner [8] trains two policies (teacher and student) that share weights but receive different privileged information (teacher sees the question plus extra context, student sees only the question). Ablations show that forward KL outperforms reverse KL because the capacity gap disappears; the teacher’s distribution is merely a sharpened version of the student’s. The work also demonstrates the necessity of per‑token KL clipping to suppress stylistic token noise.

SDPO / SRPO: Variations on Loss and Teacher Signal

SDPO [9] adds a self‑distillation loss (logit‑level KL) to GRPO’s surrogate loss on all rollouts, treating the completed rollout as a teacher for its own process. SRPO [9] builds on SDPO by applying the loss only on error trajectories (reward = 0) and reverting to vanilla GRPO on correct trajectories, reinforcing the intuition that teachers are needed only when the student errs.

RLSD and RLRT: Teacher as Magnitude Modulator and Direction Reverser

RLSD [10] uses ground‑truth answers as privileged context, turning teacher logits into per‑token advantage weights (magnitude only). RLRT [11] swaps numerator and denominator of RLSD’s weight, applies the modulation only on successful trajectories, and rewards tokens where the student deviates from the teacher’s preferred answer. This yields an 18 % gain over GRPO on six math benchmarks with Qwen‑3‑4B‑Base.

RLCSD: Canceling Style Drift with Correct‑vs‑Incorrect Contrast

Tsinghua’s RLCSD [13] identifies “privilege‑induced style drift”: teacher signals are biased toward stylistic tokens (e.g., “Therefore”) rather than task‑relevant tokens. By contrasting a correct rollout with an incorrect rollout that shares the same template, the style component cancels out, leaving a clean task signal. This improves performance across AMC23, AIME24/25, and Knights‑and‑Knaves.

SDAR: Gating OPSD in Multi‑Turn Agent Settings

SDAR [14] adapts OPSD for multi‑turn agents, where errors compound across turns and privileged context may be unreliable. It gates the OPSD signal with a sigmoid: endorsed tokens open the gate (signal amplified), rejected tokens close it (signal attenuated). This asymmetric treatment yields +7‑10 pt over GRPO on ALFWorld, WebShop, and Search‑QA.

Many Faces of OPD: When OPSD Works and When It Fails

The “Many Faces” study [15] evaluates OPSD under two privileged‑information regimes: shared‑rule PI (system prompts, style instructions) and instance‑specific PI (ground‑truth answers). OPSD excels with shared‑rule PI, achieving higher sample efficiency on CharacterBench, EmotionBench, and system‑prompt tasks, but collapses completely on instance‑specific PI (Math500, AIME24/25). The authors attribute the failure to “style leakage” that causes hallucinated references to ground‑truth answers.

TrOPD: Trust Region for Reverse‑KL Estimation

TrOPD [16] tackles the numerical instability of reverse KL on long sequences, where low‑probability teacher tokens produce outlier gradients that crash training. Inspired by speculative decoding, it defines a trust region: tokens inside the region follow reverse KL, while outliers are handled by forward KL. This switch adds ~2 pts over simple masking and aligns with Lightning OPD’s offline‑teacher‑distribution idea.

Overall Insights

Across the surveyed works, the central theme is the evolving role of the “teacher” in post‑training LLM distillation: from a full‑time supervisor (vanilla OPD) to a conditional signal that adapts to student confidence, error state, or token‑level alignment. The analyses collectively suggest that teacher signals should be dynamic, context‑aware, and often derived from the student itself rather than a static external model.

Code example

[1]