Skip to content
ICML Oral: Left-to-Right Training Boosts Diffusion LLMs' Reasoning Despite Flexible Generation

ICML Oral: Left-to-Right Training Boosts Diffusion LLMs' Reasoning Despite Flexible Generation

Tsinghua researchers show that training diffusion language models with left-to-right token unmasking and GRPO significantly improves math and code reasoning performance, reducing entropy at logical branching points and outperforming flexible decoding strategies on GSM8K and MATH-500 benchmarks.

AI Frontier Lectures
AI Frontier Lectures
AI Frontier Lectures

Paper Overview

An ICML 2026 Oral paper from Tsinghua University's LeapLab (Zanlin Ni et al., corresponding author Huang Gao) investigates whether diffusion language models (DLLMs) benefit from their inherent flexible token unmasking order during reasoning tasks. Contrary to the intuition that flexible ordering enables broader exploration, the authors find that enforcing left-to-right unmasking during reinforcement learning with Group Relative Policy Optimization (GRPO) yields higher accuracy on mathematical and code generation benchmarks.

Method: Left-to-Right Training with GRPO

The base model is LLaDA 8B Instruct. During training, the unmasking order is changed from confidence-based (arbitrary order) to strict left-to-right. The model generates sequences of length 256. GRPO samples 16 responses per question at temperature 1.0, computes per-token advantages by normalizing rewards within the group, and uses importance sampling ratios with clipping. The KL penalty coefficient is set to 0. Rewards are binary for math (exact match) and unit-test pass rate plus format score for code. Training uses full-parameter fine-tuning with AdamW (learning rate 5e-6, constant schedule, weight decay 0, global batch size 64) for 125 steps on 16 H100 GPUs, without prior supervised fine-tuning on the target tasks.

Experimental Setup

Math training uses official training sets from each dataset. Code training selects 21,000 hard examples with unit tests from AceCoder-87K. Evaluation follows LLaDA's original protocol: confidence-based unmasking with block size 32, 256 steps for 256 tokens, temperature 0. This means evaluation still uses the diffusion model's native arbitrary-order decoding, while only training uses left-to-right.

Results

On GSM8K, the left-to-right trained model achieves 89.1% vs. SPG's 86.9% (under matched conditions: full-parameter, single token per step, length 256). On MATH-500, it scores 45.1% vs. SPG's 41.8%. On HumanEval and MBPP, the method also outperforms baselines (d1, ESPO, SPG, GDPO). The improvement is consistent across benchmarks.

Analysis: Entropy Reduction at Branching Points

Figure 7 shows average entropy when unmasking logical connectives (Therefore, Thus, Since, To, Now, determine, Given). Under arbitrary-order decoding, these words are deferred until later, so their entropy drops because subsequent context constrains the choices ("entropy collapse"). Under left-to-right decoding, no future context exists, so entropy remains high at these branching points. The overall average entropy across all tokens is similar between the two orders, confirming that the reduction is localized to decision points, not a vocabulary-wide effect.

Entropy at branching words: arbitrary order (blue) vs left-to-right (orange)
Entropy at branching words: arbitrary order (blue) vs left-to-right (orange)

Unique Problem Coverage

Venn diagrams (1024 samples, base LLaDA-Instruct) reveal that left-to-right decoding solves more unique problems, especially in code. On HumanEval, 21.3% of solved problems are exclusive to left-to-right vs. 0.6% for arbitrary order. On GSM8K, the gap is 1.2% vs. 0.0%. The arbitrary-order solved set is largely a subset of the left-to-right set, indicating that flexible decoding does not explore meaningfully different solution paths; it merely defers difficult decisions.

Venn diagrams of unique solved problems per benchmark
Venn diagrams of unique solved problems per benchmark

Ablation Studies

Block size controls the freedom of arbitrary-order decoding (block size 1 = pure left-to-right). On HumanEval, Pass@k decreases as block size increases (tested at 8, 32, 128). Default block size is 32. A controlled comparison on GSM8K shows: base model 78.6%, random-order training 82.2%, left-to-right training 89.1%, proving the benefit comes from the specific left-to-right order, not merely fixing any order. Additionally, after left-to-right training, the model degrades more gracefully when evaluated with larger unmasking steps (multiple tokens per step) compared to the base model.

Accuracy vs tokens unmasked per step: base vs trained model
Accuracy vs tokens unmasked per step: base vs trained model

Conclusion

Forcing left-to-right unmasking during GRPO training improves diffusion LLMs' reasoning by preventing premature entropy collapse at logical branching points. The model learns to make high-entropy decisions early, guided by the RL signal, rather than deferring them to later stages where they become artificially easy. This yields substantial gains on math and code benchmarks without architectural changes.

References

Paper: https://arxiv.org/abs/2601.15165

Code: https://github.com/LeapLabTHU/JustGRPO

Code example

👀点击关注
AI前沿快速Follow
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

code generationGRPOEntropy ReductionMathematical ReasoningICML 2026diffusion language modelsLLaDALeft-to-Right Training
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.