ICML Oral: Left-to-Right Training Boosts Diffusion LLMs' Reasoning Despite Flexible Generation
Tsinghua researchers show that training diffusion language models with left-to-right token unmasking and GRPO significantly improves math and code reasoning performance, reducing entropy at logical branching points and outperforming flexible decoding strategies on GSM8K and MATH-500 benchmarks.
Paper Overview
An ICML 2026 Oral paper from Tsinghua University's LeapLab (Zanlin Ni et al., corresponding author Huang Gao) investigates whether diffusion language models (DLLMs) benefit from their inherent flexible token unmasking order during reasoning tasks. Contrary to the intuition that flexible ordering enables broader exploration, the authors find that enforcing left-to-right unmasking during reinforcement learning with Group Relative Policy Optimization (GRPO) yields higher accuracy on mathematical and code generation benchmarks.
Method: Left-to-Right Training with GRPO
The base model is LLaDA 8B Instruct. During training, the unmasking order is changed from confidence-based (arbitrary order) to strict left-to-right. The model generates sequences of length 256. GRPO samples 16 responses per question at temperature 1.0, computes per-token advantages by normalizing rewards within the group, and uses importance sampling ratios with clipping. The KL penalty coefficient is set to 0. Rewards are binary for math (exact match) and unit-test pass rate plus format score for code. Training uses full-parameter fine-tuning with AdamW (learning rate 5e-6, constant schedule, weight decay 0, global batch size 64) for 125 steps on 16 H100 GPUs, without prior supervised fine-tuning on the target tasks.
Experimental Setup
Math training uses official training sets from each dataset. Code training selects 21,000 hard examples with unit tests from AceCoder-87K. Evaluation follows LLaDA's original protocol: confidence-based unmasking with block size 32, 256 steps for 256 tokens, temperature 0. This means evaluation still uses the diffusion model's native arbitrary-order decoding, while only training uses left-to-right.
Results
On GSM8K, the left-to-right trained model achieves 89.1% vs. SPG's 86.9% (under matched conditions: full-parameter, single token per step, length 256). On MATH-500, it scores 45.1% vs. SPG's 41.8%. On HumanEval and MBPP, the method also outperforms baselines (d1, ESPO, SPG, GDPO). The improvement is consistent across benchmarks.
Analysis: Entropy Reduction at Branching Points
Figure 7 shows average entropy when unmasking logical connectives (Therefore, Thus, Since, To, Now, determine, Given). Under arbitrary-order decoding, these words are deferred until later, so their entropy drops because subsequent context constrains the choices ("entropy collapse"). Under left-to-right decoding, no future context exists, so entropy remains high at these branching points. The overall average entropy across all tokens is similar between the two orders, confirming that the reduction is localized to decision points, not a vocabulary-wide effect.
Unique Problem Coverage
Venn diagrams (1024 samples, base LLaDA-Instruct) reveal that left-to-right decoding solves more unique problems, especially in code. On HumanEval, 21.3% of solved problems are exclusive to left-to-right vs. 0.6% for arbitrary order. On GSM8K, the gap is 1.2% vs. 0.0%. The arbitrary-order solved set is largely a subset of the left-to-right set, indicating that flexible decoding does not explore meaningfully different solution paths; it merely defers difficult decisions.
Ablation Studies
Block size controls the freedom of arbitrary-order decoding (block size 1 = pure left-to-right). On HumanEval, Pass@k decreases as block size increases (tested at 8, 32, 128). Default block size is 32. A controlled comparison on GSM8K shows: base model 78.6%, random-order training 82.2%, left-to-right training 89.1%, proving the benefit comes from the specific left-to-right order, not merely fixing any order. Additionally, after left-to-right training, the model degrades more gracefully when evaluated with larger unmasking steps (multiple tokens per step) compared to the base model.
Conclusion
Forcing left-to-right unmasking during GRPO training improves diffusion LLMs' reasoning by preventing premature entropy collapse at logical branching points. The model learns to make high-entropy decisions early, guided by the RL signal, rather than deferring them to later stages where they become artificially easy. This yields substantial gains on math and code benchmarks without architectural changes.
References
Paper: https://arxiv.org/abs/2601.15165
Code: https://github.com/LeapLabTHU/JustGRPO
Code example
👀点击关注
AI前沿快速FollowSigned-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
