Skip to content
STOPD: Lossless OPD Acceleration via Random Truncation and Inverse Probability Weighting

STOPD: Lossless OPD Acceleration via Random Truncation and Inverse Probability Weighting

STOPD introduces random truncation with inverse probability weighting and block-wise stopping to accelerate on-policy distillation, reducing generated tokens by 25-48% and achieving 1.3-2.2x training speedup while maintaining accuracy comparable to vanilla OPD, with theoretical guarantees of unbiased gradient estimation and error bounds for SGD, Adam, and Muon optimizers.

Machine Heart
Machine Heart
Machine Heart

Introduction

On-policy distillation (OPD) lets a student model generate responses and a teacher provide per-token feedback, enabling direct guidance on the student's own outputs. However, each training round requires the student to generate a full response and the teacher to evaluate every token, causing cost to grow with response length.

Existing Approaches to Reduce OPD Overhead

Two main strategies exist:

Sparse supervision : The student still generates a full response, but only a subset of tokens are selected for training. Methods differ in selection criteria: TIP uses student uncertainty and teacher-student disagreement; TA-OPD considers teachability of teacher signals; IER focuses on gradient estimation reliability. All reduce the number of supervised tokens.

Direct truncation : Generation is stopped early. Early Stopping Rollout (ESR) keeps a fixed-length prefix. Progressive OPD (POPD) gradually increases generation length with training progress. Prune-OPD adjusts suffix weights and generation budget based on predicted overlap. KAT stops early when teacher-student KL divergence stays low. Truncation saves computation but discards gradient contributions from truncated positions, biasing the expected gradient unless compensated.

STOPD Method

STOPD combines three components:

Random truncation : Before each response, independently sample a uniform stop position. Generation halts at that position unless the student ends naturally earlier. No further tokens are generated, and no teacher feedback or loss is computed for truncated parts.

Inverse probability weighting (IPW) : To correct the sampling bias, each retained token's loss is multiplied by the inverse of its retention probability p. This compensates for the fact that earlier positions are retained more often than later ones.

Block-wise truncation : To prevent extremely large weights for tail tokens (e.g., weight 16,384 for the last token at max length 16,384), positions are divided into equal-width contiguous blocks. When a position is sampled, generation continues to the end of its block. This raises retention probabilities for later positions within a block and caps the maximum weight. For max length 16,384 and block width 128, the maximum weight drops from 16,384 to 128.

The following animation illustrates random truncation and the weighted contribution of each position:

Animation of random truncation and per-position weighted contributions
Animation of random truncation and per-position weighted contributions

Why the Gradient Sum Is Unbiased

Consider a token with gradient contribution g in a full response. Suppose it has a 1/4 probability of being retained after the random stop. Under STOPD, if retained its contribution becomes 4g; otherwise 0. The expectation over the random stop position is (1/4)*4g + (3/4)*0 = g, matching the full-generation contribution. Summing over all positions recovers the expected gradient sum of the complete response.

Illustration of unbiased gradient expectation with IPW
Illustration of unbiased gradient expectation with IPW

The paper further analyzes loss normalization in actual training and derives single-step update error bounds for SGD, Adam, and Muon optimizers.

Main Experimental Results

Experiments use Qwen3-4B-Instruct-2507 as teacher and Qwen3-1.7B-Base as student on math, code, and instruction-following tasks. STOPD and vanilla OPD show similar distillation loss curves and final evaluation results:

Math: vanilla OPD 30.1% accuracy vs STOPD 29.4%; training speedup 1.8x , token reduction 34% .

Code: vanilla OPD 28.5% pass rate vs STOPD 28.1%; speedup 2.2x , token reduction 48% .

Instruction following: vanilla OPD 59.8% accuracy vs STOPD 60.1%; speedup 1.3x , token reduction 25% .

Accuracy, training speedup, and training dynamics across three domains
Accuracy, training speedup, and training dynamics across three domains

Larger Models

With teacher Qwen3-30B-A3B-Instruct-2507 and student sizes 4B and 8B, STOPD maintains accuracy close to vanilla OPD with similar loss curves. Token reduction: 35% (4B) and 33% (8B); speedup: 1.8x (4B) and 1.7x (8B).

Results on 4B and 8B student models
Results on 4B and 8B student models

Different Optimizers

On math tasks, replacing AdamW with SGD and Muon still yields speedups of 2.2x (SGD) and 1.9x (Muon) with comparable accuracy: SGD 25.2% vs 25.1%; Muon 29.5% vs 29.8%.

Accuracy and training dynamics under AdamW, SGD, and Muon
Accuracy and training dynamics under AdamW, SGD, and Muon

Batch Size and Block Width Ablation

Random truncation introduces extra loss variance. Increasing batch size from 128 to 512 reduces loss jitter from 12.2% to 6.0% while preserving 2.0-2.1x speedup. The near-halving of jitter with 4x batch size matches the theoretical 1/sqrt(batch size) scaling. Block width sweeps from 32 to 2048 show similar accuracy; default block width 128 is used.

Effect of batch size and block width on loss jitter and accuracy
Effect of batch size and block width on loss jitter and accuracy

Comparison with Fixed Truncation (ESR)

On math tasks, ESR's accuracy plateaus early while evaluation responses keep growing longer, hitting the length limit more often. STOPD's accuracy and stopping behavior stay closer to vanilla OPD. ESR's fixed cutoff prevents supervision of later tokens, including natural end-of-sequence tokens, so the student never learns when to stop. STOPD gives every position a chance to be trained and uses IPW to correct its lower retention probability.

Comparison of ESR, vanilla OPD, and STOPD on accuracy, response length, and length-limit hit rate
Comparison of ESR, vanilla OPD, and STOPD on accuracy, response length, and length-limit hit rate

Conclusion

OPD does not require generating to the end every time. By randomly choosing truncation points and compensating sampling bias with inverse probability weighting, STOPD reduces generation compute while preserving the expected gradient contribution of each token. The paper and code are publicly available. The same principle may benefit other per-token supervision methods such as online/offline knowledge distillation and supervised fine-tuning.

Paper: https://arxiv.org/abs/2610.21337 Code: https://github.com/YouAreSpecialToMe/STOPD Hugging Face:

https://huggingface.co/papers/2610.21337
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Large Language ModelsQwenTraining AccelerationOn-Policy DistillationGradient EstimationInverse Probability WeightingRandom TruncationSTOPD
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.