12K GPU-Hour Reality Check: GRPO Outperforms 10 Self-Distillation Methods in Agentic RL

Background: Reproducibility Crisis in Agentic RL Self-Distillation

Researchers from Zhejiang University and Ant Group encountered difficulties reproducing reported gains from self-distillation methods in Agentic Reinforcement Learning. Papers often used inconsistent baselines, training environments, and evaluation protocols, making it unclear how much of the reported improvement came from the method itself.

Unified Experimental Design

The team compared 11 training methods (GRPO plus 10 self-distillation variants) across 4 task categories (ALFWorld, ScienceWorld, Search-QA, WebShop) and 3 model sizes (Qwen 2B, 4B, 9B). Each method was trained for 100 steps on each of the 12 model-task combinations, with validation every 5 steps. Final models were evaluated with 8 random seeds. The study consumed 12,124.4 H800 GPU hours for training, validation, and checkpointing (excluding repeated evaluations).

Key Finding: No Universal Winner Over GRPO

Across the 12 settings, 10 self-distillation methods each had some wins, but none exceeded GRPO in every setting. For example, Skill-SD achieved 74.29% success on ALFWorld with 2B (vs. GRPO's 28.48%), but scored 0% on WebShop with 4B and 9B (vs. GRPO's 66.02% and 70.02%). Averaging equally across all 12 settings, PCSD ranked first (54.26), AgentOPSD second (51.34), and GRPO third (49.68). Only PCSD and AgentOPSD surpassed GRPO on average. PCSD beat GRPO in 6 settings, lost in 5, tied in 1, yet never placed first in any single setting's overall metric. ICSD won in 7 settings but averaged only 44.75 due to a catastrophic 0% on 4B WebShop.

Training Instability Hidden by Final Scores

Ranking methods by validation AUC (area under the curve) during training produced a different order than final scores. OCSD led in AUC but finished 4th in final score; AgentOPSD finished 2nd in final score but 9th in AUC. In the 4B ALFWorld setting, AgentOPSD reached 60% at step 35, then collapsed below 4.3% for six consecutive validations before recovering to 79.3% at step 100. GRPO+OPSD never recovered from a similar collapse (ending at 2.1%), while OCSD peaked at 85% then degraded. These patterns show that training stability and the ability to retain gains are critical but often unreported.

Decoding Strategy Drastically Changes Reported Performance

On 2B Search-QA, OVCSD's final model scored 1.34% with greedy decoding but 33.21% with temperature 0.4 sampling (8 seeds). The same test set yields vastly different numbers depending on evaluation protocol, a factor rarely controlled in cross-paper comparisons.

Compute-Equivalent Comparisons Reverse Apparent Gains

Self-distillation methods require extra teacher scoring, trajectory sampling, or advantage computation. In 120 pairwise comparisons with GRPO at 100 steps, 112 needed more wall-clock time; of those, 58 still scored lower and 7 tied. PCSD and AgentOPSD averaged 17% and 19% longer training. When the time budget was capped at each setting's GRPO training duration, AgentOPSD's ALFWorld advantage (68.10% vs. GRPO's ~58% at 100 steps) vanished, dropping to 37.63%. PCSD's WebShop lead similarly disappeared under equal compute.

Conclusion and Open Resources

Self-distillation can yield large gains in specific conditions, but no method reliably dominates GRPO across tasks, model sizes, and compute budgets. The authors release training and evaluation code for all 11 methods at https://github.com/aim-uofa/Self-Distillation-AgentRL and the paper at https://arxiv.org/abs/2610.19789, providing a reusable benchmark for future Agentic RL research.