Unveiling Scaling Laws for Mixture-of-Experts Diffusion Language Models: LLaDA MoE v2

Diffusion language models have become a prominent direction in large‑language‑model research due to their bidirectional context modeling and parallel token decoding. As model size and training compute grow, the community needs principled scaling laws to answer how optimal hyper‑parameters, compute allocation, and architecture should evolve.

Existing scaling laws for autoregressive models cannot be directly transferred because diffusion models use a masked‑denoising objective, where supervision is only applied to masked tokens and each prediction depends on a corrupted sequence. Consequently, the authors systematically characterize scaling behavior specific to mixture‑of‑experts (MoE) diffusion models.

By jointly searching batch size and learning rate across model scales from 158 M to 3.6 B parameters and FLOP budgets, they fit a diffusion‑specific hyper‑parameter law: the optimal batch size grows more steeply with compute than in autoregressive models, while the optimal learning rate decays faster. For a given FLOP budget, the autoregressive law predicts an optimal batch size of about 1.02 M tokens, whereas the new law suggests 3.43 M tokens, reflecting the reduced effective supervision in masked‑denoising training. Extrapolations of this law match measured optimal configurations.

The authors also perform an IsoFLOP analysis to allocate a fixed compute budget between model size and training tokens. They find that the optimal model‑side compute scales as the total FLOPs to the 0.475 power, while the optimal token count scales as the total FLOPs to the 0.525 power, indicating a slight bias toward data. Compared with dense diffusion models, the MoE frontier is the most data‑biased observed, highlighting the combined “data‑friendly” effect of sparse activation and diffusion training.

For architecture design, three dimensions are examined: activation ratio, expert granularity, and shared‑expert proportion. As scale increases, lower activation ratios (i.e., larger expert pools) become optimal; a medium granularity of 8–16 experts per token remains robust across scales; and a shared‑expert proportion of roughly 33.3 % consistently yields the best performance, contrasting with the 25 % or no‑shared‑expert settings common in autoregressive MoE models.

Guided by these three scaling laws, the team trained LLaDA MoE v2, a 30B‑parameter diffusion model with 3B active parameters per token. The architecture uses 128 fine‑grained routing experts, a 9.09 % activation ratio, granularity of 8, and a 33.3 % shared‑expert share—settings that lie within the law‑predicted optimal region. Training consumed 23.5 T tokens (≈ 46 × 10⁴ NVIDIA B200 GPU‑hours).

On a suite of 15 benchmarks, LLaDA MoE v2 achieves the highest average diffusion‑model score of 58.60, surpassing the previous best diffusion model SDAR Sci by 3.78 points. It leads HumanEval and BigCodeBench by 16.46 and 7.98 points respectively, and narrows the gap to the autoregressive Qwen3‑30B model to less than 3 points on several reasoning and code tasks, while using only about 65 % of Qwen3’s pre‑training tokens.

Compared with the previous LLaDA‑MoE 7B‑A1B, the new model reaches higher scores on MMLU, GSM8K, and KorBench with roughly half the training compute, illustrating the practical efficiency of law‑guided scaling.

For supervised instruction fine‑tuning, the team used 7 M instruction examples. The fine‑tuned model outperforms the same‑size diffusion baseline SDAR Chat on 7 of 8 math‑reasoning and code generation benchmarks, approaches Qwen3 on AIME 2024/2025, MBPP, LiveCodeBench, and surpasses it on the multilingual code benchmark MultiPL‑E.

In summary, LLaDA MoE v2 establishes the first systematic scaling laws for mixture‑of‑experts diffusion language models, enabling law‑guided sparse architecture and efficient compute allocation. The resulting model achieves competitive performance with substantially reduced pre‑training tokens and activation cost, paving the way for further scaling of diffusion‑based LLMs.