Nvidia Cosmos 3: One Model Handles Physical AI Perception, Reasoning, Action, and Simulation

Physical AI requires an intelligence that can simultaneously understand visual scenes, interpret audio, process language, predict motion, generate actions, and integrate all of these capabilities. Nvidia’s Cosmos 3 delivers this by providing a single model that handles five modalities—text, image, video, audio, and motion—within a unified Mixture‑of‑Transformers (MoT) framework.

Five Modalities, One Architecture

Previously, developers assembled separate models for each task: Cosmos Predict for world generation, Cosmos Transfer for controlled generation, Cosmos Reason for scene understanding, and Cosmos Policy for strategy generation. This fragmented approach required four models, four inference pipelines, and incurred high switching costs and duplicated representations, preventing information flow between models. Cosmos 3 consolidates these roles into a single framework, eliminating the need for separate encoders and allowing shared representations of object position, motion, and sound.

Each modality is first processed by its own encoder—ViT for visual understanding, VAE for visual and audio generation, and domain‑aware vectors for motion—then projected into a shared representation space. The MoT architecture shares most computation across modalities, branching only where necessary, which dramatically improves parameter efficiency; a 16 B‑parameter Nano model can match the performance of several specialized models.

Input sequences are split into two sub‑sequences: an autoregressive (AR) branch for reasoning and understanding via next‑token prediction, and a diffusion (DM) branch for generation via iterative denoising. Both branches run in each Transformer layer with separate parameter sets but interact through Joint Attention, enabling the AR branch to guide generation and the DM branch to validate reasoning, forming a closed‑loop "reasoning + generation" coupling.

Dual‑Line Advantage

The coupled inference‑generation pipeline allows tasks such as "place a flower in a red bottle" to be solved by first reasoning the grasp trajectory in the AR channel and then generating a corresponding video in the DM channel. This "think‑then‑act" approach yields more controllable outputs and lower error rates than end‑to‑end generation.

Cosmos 3 excels on both reasoning and generation benchmarks: it ranks first among open‑source models on robot, intelligent‑space, and autonomous‑driving reasoning tests, and also leads in text‑to‑image, text‑to‑video, and robot‑policy generation. Evaluations by Artificial Analysis and RoboArena confirm its superiority.

Two Sizes, Fully Open‑Source

Cosmos 3 is released under the Linux Foundation’s OpenMDW‑1.1 license with all code, model weights, curated synthetic datasets, and evaluation benchmarks publicly available. Two specifications are offered:

Nano : 16 B parameters (8 B inference + 8 B generation), optimized for efficient inference on a single RTX PRO 6000 GPU, suitable for edge deployment such as real‑time factory sorting robots.

Super : 64 B parameters (32 B inference + 32 B generation), targeting large‑scale data generation and research on Nvidia Hopper or Blackwell GPUs.

Six high‑quality synthetic datasets covering Physical‑Interaction‑Scenes, Embodied‑Robot‑Scenes, Autonomous‑Driving‑Scenarios, Warehouse‑Operations‑Scenes, and more are provided via Hugging Face, reducing the barrier for training and evaluating world models. Post‑training scripts and the Agent Skills toolkit facilitate fine‑tuning for specific robots, environments, or tasks.

Overall, Cosmos 3 demonstrates how a unified MoT architecture can share computation between perception and generation, use joint attention to couple reasoning with synthesis, and deliver a complete open‑source stack for Physical AI research and development.

References: https://research.nvidia.com/labs/cosmos-lab/cosmos3/, https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf, https://github.com/nvidia/Cosmos, https://huggingface.co/blog/nvidia/cosmos-3-for-physical-ai