Nvidia’s First Tri‑Mode LLM Boosts Token Throughput 4× and Promises Second‑Second Long‑Text Generation

Nvidia presents the world’s first tri‑mode large language model (LLM) that unifies autoregressive (AR), diffusion, and self‑speculation decoding within a single architecture, requiring only a simple attention‑mask change to toggle modes and no additional draft models or architectural modifications.

Motivation

Traditional AR decoding suffers from memory‑bound token generation at low batch sizes, limiting GPU utilization and response speed for single‑user AI assistants. Diffusion models offer parallel generation but historically lag in quality due to the lack of left‑to‑right language priors.

Unified Design

The proposed model combines the strengths of both paradigms: it drafts multiple tokens in diffusion mode using a block‑wise denoising process with dual‑stream attention, then validates them in AR mode with the same KV cache, achieving diffusion‑level parallelism without sacrificing AR accuracy.

Three Decoding Modes

AR Mode: Standard left‑to‑right token generation with full causal attention, suited for high‑concurrency cloud services.

Diffusion Mode: Block‑wise denoising with dual‑stream attention and a lightweight trained sampler replaces conventional confidence thresholds, enabling massive parallel token speculation.

Self‑Speculation Mode: Replaces the external small draft model of conventional speculative decoding with a single‑model self‑competition mechanism.

Training Objective

The model optimizes both AR loss and diffusion loss simultaneously. To stabilize training, Nvidia employs a two‑stage schedule and introduces Global Loss Averaging, which mitigates gradient spikes caused by random masking in diffusion training.

Model Variants and Accuracy

Three base model sizes (3B, 8B, 14B) are released. Compared with open‑source dLLMs such as LLaDA, Dream, and SDAR, they improve accuracy by 9 %–22.4 %, establishing a new state‑of‑the‑art for diffusion LLMs.

Performance Benchmarks

DGX Spark (FP8): 3.14× speedup (112 tok/s vs 41.8 AR); INT4: 2.7×.

RTX 6000 Pro (FP8): 3.4×; INT: 2.3×.

GB200: 3.3× (850 tok/s); with custom CUDA kernels up to 4×.

On the SPEED‑Bench suite, linear self‑speculation achieves an average acceptance length of 8.7, compared to 4.7 for Qwen3.5‑9B‑MTP and 2.81 for Qwen3‑8B‑Eagle3.

Scalability and Deployment

At low‑to‑moderate concurrency, self‑speculation dominates, ideal for personal AI agents. For massive batch sizes (>64 streams), the system reverts to pure AR mode to avoid compute bottlenecks, ensuring efficient operation across all deployment scenarios.

Training Recipe

The full training pipeline includes 1 trillion tokens of AR‑only pre‑training, followed by 300 billion tokens of joint AR + diffusion training, and subsequent SFT and VLM alignment.

Key Technical Innovations

Global loss averaging with DP‑rank dynamic masking.

Strict causal clean flow to prevent label leakage.

LoRA‑enhanced drafter for improved self‑speculation.

Future Outlook

The authors argue that future LLM architectures should not force a choice between AR and diffusion; instead, integrating both within a single transformer may be the optimal path. They estimate that a perfect diffusion sampler could raise diffusion mode performance by an additional 76.5 % over current self‑speculation, bringing “second‑second” long‑text generation closer to reality.