NVIDIA's AW-0: Agentic World Model Unifies Code, Video & Action for Robot Control

Architecture: One Backbone, Three Experts

AW-0 centers on a Mixture-of-Transformers backbone hosting three specialized experts that co-evolve during every denoising iteration:

Code Expert : Built on a pretrained language backbone with a frozen Qwen3-VL ViT for vision; generates skill programs via LLaDA-style masked diffusion.

Video Expert : Initialized from Cosmos-Predict2.5 video DiT; predicts future frames using continuous flow matching.

Action Expert : An action DiT outputting 64-dimensional continuous action chunks.

The three experts are not independent — they jointly evolve in each denoising step, allowing semantic plans to condition video and action generation throughout.

Data Pipeline: Automatic Annotation with Agent Teams

Training requires massive aligned video + program pairs. AW-0 automates this with an agent pipeline:

For each task, one demonstration is sampled; Gemini 3.1 Pro writes a subtask-level program using a declarative primitive API.

Claude Code (Claude Opus 4.8) acts as coordinator, distilling recurring patterns into reusable skills frozen into a skill library (23 YAM robot skills + 27 first-person human skills = 50 total).

During remaining corpus annotation, subtask contact-and-motion patterns hit the cache for direct reuse; misses trigger new annotation and cache write-back.

Simulation Evaluation: RoboTwin 2.0 Leadership

On RoboTwin 2.0 (50 tasks, 50 rollouts each), AW-0 achieves 50.7% average success rate , ranking first. Two notable details:

Clean setting: 80.8% vs. strongest baseline WLA ( +16 pp ).

Visual randomization: hybrid execution mode lifts success from 20.6% to 29.2% , surpassing all baselines — code execution robustness compensates for pixel-level out-of-distribution prediction gaps.

Real-Robot Evaluation: Dual-Arm & Dexterous Hand

Experiments cover two platforms: parallel-gripper YAM dual-arm station and 22-DoF Sharpa dexterous hand on Tianji–Sharpa, each with 4 tasks, 3×16 rollouts per task. AW-0 scores 71.1% average , +25.1 pp over the strongest baseline, winning all 8 tasks.

Language following & semantic reasoning : Arrange Fruits (91.7%) and Flip Scoreboard (68.8%) show code planning injects semantic capability into the world-action model.

Dexterous manipulation : On Sharpa, baselines drop to 20–30%; AW-0 reaches 63.4% . Solve Cube hits 72.1% vs. best baseline 52.3% — skill programs directly enable long-horizon fine-grained action decomposition.

Ablation: Three Design Decisions Validated

Attention order : Code-first (C→V→A) works best — video and action generation attend to semantic plans throughout, not just post-hoc reasoning.

Annotation content : Full programs (comments + skill calls) beat no-comment code and natural-language subgoals on all 8 tasks. Removing skill calls drops Deal Poker from 58.3% to 42.0% , proving reusable skill abstractions add value.

Generation paradigm : Joint denoising beats autoregressive (AR) on 7/8 tasks ( 71.1% vs 54.5% ). Inference latency drops from 0.55 s to 0.23 s on RTX 5090 (2.39× speedup). Compared to a no-code-expert baseline (0.19 s), adding the code expert costs only 0.04 s for full performance gains.

Paper: https://arxiv.org/pdf/2610.30136 Project page: https://agentic-wam.github.io/