One Problem, 70% Gains: Rethinking On-Policy Distillation's Data Efficiency

Extreme Data Reduction in On-Policy Distillation

Researchers from Tsinghua University, Chinese Academy of Sciences, Northeastern University, UIUC, and Johns Hopkins reduced the OPD training set from 17,000 math problems (DAPO-Math-17K) to a single problem. Contrary to expectations that the model would quickly exhaust the single example, performance continued improving for hundreds of steps. On three math benchmarks, the single-problem setup recovered 87% of the full-data OPD gain at 300 steps (average score 68.5 vs. 69.8 for full data), starting from 59.1. At 1,000 steps, the single-problem score reached 68.4 versus 72.1 for full data (72% recovery). The gap stabilized around 3 points after 300 steps, indicating a persistent ceiling.

Generalization Across Tasks and Model Families

The phenomenon held across coding, instruction following, and agentic tool use, where single-problem training closed 73%, 66%, and 64% of the initial teacher-student gap respectively. Experiments with Qwen, Llama, and OLMo teacher-student pairs confirmed robustness.

Insensitivity to Problem Difficulty and Hyperparameters

Three math problems of varying difficulty yielded similar gains. The hardest problem was never solved by the student during 300 steps, yet validation performance still rose. Varying response length limits and sampling temperature produced comparable improvements.

Data Side: State Coverage Explains High Gains from One Problem

OPD's learning unit is not the problem but the state — the problem plus the partially generated response. Each update contains 64 rollouts, yielding many supervised token positions even from one problem. The authors define state coverage : clustering all states visited by full-data OPD into 200 reference clusters, then measuring what fraction a given setup visits. One problem covered 65.9% of reference clusters at 100 steps and 71.5% at 300 steps; additional 200 steps added only 5.6 percentage points.

Controlled experiments isolated the effect of state diversity:

Rollout-side: Fixing 1, 4, 16, or 64 trajectories from the initial student and repeating them to fill the batch. Validation accuracy rose from 0.606 to 0.666 as trajectory diversity increased.

Problem-side: Selecting 1, 4, or 16 problems from different semantic clusters. State coverage reached 71.5%, 79.8%, and 98.9% respectively; 16 diverse problems matched full-data validation performance under the same compute budget.

Diversity vs. quantity: Two sets of 16 problems — one from 16 clusters, one from a single cluster. The diverse set achieved higher coverage and performance; the single-cluster set plateaued at ~76.8% coverage after ~100 steps.

Conclusion: In this setting, 16 semantically diverse problems saturate the usable supervision — data is "stuffed" (数据撑死).

Algorithm Side: Absorption Rate Independent of Data Quantity

State coverage describes available supervision, not how fast the student absorbs it. The authors define two dynamic metrics:

Distance (d): Mean absolute log-probability difference between teacher and student on tokens the student actually generates at its current visited positions.

Absorption rate (v): Fraction of remaining distance closed per update.

Key observations:

Distance decreases monotonically across the entire run — the student never stalls. However, absorption rate declines steadily, explaining why hundreds of steps are needed.

The absorption-rate curve is nearly identical for 1, 4, 16, and 17k problems. At 300 steps, each run had closed 78%-84% of its own initial distance. Data quantity does not alter the alignment rhythm.

To test whether continuous generation of new states causes the slow absorption, a fixed-trajectory control reused the initial student's 64 rollouts throughout training. Validation performance still improved for ~200 steps, proving that slow absorption of existing supervision is intrinsic , not just due to shifting state distributions.

Learning-rate sweeps (1x, 2x, 4x) showed that higher rates compress the timeline but, when rescaled by learning rate, the distance curves overlap — the shape of absorption decay is invariant.

Extension to Multi-Teacher OPD (MOPD)

In a multi-domain MOPD setup (math, code, general), using 16 diverse problems per domain matched full MOPD performance. Initial average score 43.5; 300-step scores: 1 problem per domain = 50.1, 16 diverse problems = 52.9, full MOPD = 52.8 (three-teacher average 55.4).

Extreme Experiments: Training Without Meaningful Task Content

Following the "state is the unit" insight, the team tested inputs with no real task:

Empty template: User message empty, immediately followed by <think>.

System template: Only a domain instruction line.

WildChat prompts: 192,824 real user chats, only 0.17% math-related.

All three tracked the real-data baseline closely while using one-third to one-half the rollout tokens. In code, real data consumed 277M rollout tokens over 500 steps; empty template used only 18M. However, if the template closes <think> immediately, the student produces short meta-responses and fails to generate usable trajectories. Thus, task content is not the sole source of useful supervision; the input must also enable rollouts that reach states where the teacher can provide effective guidance.

Controlled Comparison: OPD vs. RLVR on a Single Problem

Running both for 1,000 steps on one problem, OPD's validation gain was over twice that of RLVR, and OPD closed 72% of the teacher-student gap. However, the ceilings differ: RLVR is not bound by a teacher's distribution, while OPD is trained to match one. RLVR is limited by outcome-based signal diversity; OPD is limited by the speed of absorbing dense supervision.

Conclusions and Practical Implications

The experiments do not imply training data is unimportant. Rather, in OPD, problem count is a poor proxy for effective supervision volume . A single problem generates broad state coverage via repeated rollouts. The bottleneck is the student's absorption speed. For data selection, the key questions become: Does this problem take the student to unseen states? Does the teacher have something to teach in those states? Can the student actually absorb that supervision within the training budget?

The two Rethinking OPD papers converge: dense rewards in OPD are not a free lunch. The scarce resource is never the supply — first paper: a higher-scoring teacher isn't necessarily a better teacher; this paper: more data isn't necessarily what the student can currently eat. The "state" perspective may generalize to other on-policy training paradigms.