Nvidia Cosmos 3: One Model Replaces Four Physical AI Systems and Unifies Five Modalities (10K+ Stars)

Fragmented pipelines in physical AI

Building robot or autonomous‑driving physical AI systems traditionally requires separate models for visual understanding, video generation, physics simulation, and motion control, leading to high switching costs, duplicated computation, and poor real‑time inference on edge devices.

Cosmos 3 breakthrough

Nvidia’s newly released Cosmos 3 unifies the five major modalities—language, image, video, audio, and action—into a single model, achieving top rankings on multiple open‑source leaderboards and outperforming other open models on the RoboArena robot‑strategy benchmark.

Core architecture

The model adopts a Mixture‑of‑Transformers (MoT) dual‑tower design that merges the previous four‑model split architecture (Cosmos Predict, Transfer, Reason, Policy) into one shared framework, removing the need for manual data flow between isolated pipelines.

Five‑modality unified representation : Text, image, video, audio, and action inputs are encoded by dedicated encoders (ViT for vision, VAE for generation, domain‑aware vectors for motion) and projected into a common shared space.

Parameter efficiency : The 16B Nano version matches the capability of multiple specialized models while using far fewer parameters.

Dual‑chain processing

The shared representation is split into two cooperating sub‑sequences:

AR (autoregressive) sub‑sequence : Handles understanding, reasoning, and planning using next‑token prediction for scene parsing, physical causality, and trajectory inference.

DM (diffusion) sub‑sequence : Handles generation, simulation, and replication using iterative denoising for images, videos, audio, and motion sequences.

Both sub‑sequences run in parallel within each Transformer layer and interact through joint attention , forming a closed loop where reasoning guides generation and generated results validate reasoning. For example, the command “place the flower in the red vase” first triggers the AR chain to infer grasp coordinates and motion trajectory, then the DM chain generates a video of the action and cross‑checks physical plausibility.

Forward and inverse dynamics

Forward dynamics (future prediction) : Given current frames, sensor data, and control signals, the model predicts future video frames and motion trajectories, useful for autonomous‑driving scenario preview and robot motion simulation.

Inverse dynamics (action reverse‑engineering) : From an operation video, the model reconstructs the underlying motion trajectory and control commands, enabling robots to learn tasks from a single human demonstration without hand‑coded control logic.

Capability overview

Cosmos 3 covers the full spectrum of physical‑AI tasks, supporting both understanding and generation:

World understanding (reasoning side)

Deep scene parsing: object detection, spatial relationships, risk identification, and chain reasoning for autonomous‑driving decisions.

Action‑sequence analysis: precise temporal labeling of robot operation videos for behavior assessment.

Physical plausibility judgment: ensures generated content respects real‑world physics.

World generation (generation side)

Text‑to‑image / image‑to‑video with preserved physical layout and material details.

Audio‑video synchronized generation: 48 kHz stereo AAC, frame rates 10‑30 FPS, resolutions 256p‑720p, frame counts 5‑300.

Conditional generation: video continuation, action‑conditioned synthesis, style transfer.

Action modeling (robot / autonomous‑driving core) Supports robot policy generation, camera motion control, vehicle trajectory planning, and is ready for embodied‑intelligence, autonomous‑driving, warehouse robotics, and industrial sorting scenarios.

Model family

Cosmos 3 Edge – 4 B parameters, ultra‑light edge version for small robots and real‑time inference on low‑power embedded GPUs.

Cosmos 3 Nano – 16 B (8 B inference + 8 B generation), general‑purpose high‑efficiency version; runs on a single RTX PRO 6000 or RTX 5090.

Cosmos 3 Super – 64 B (32 B inference + 32 B generation), flagship version for large‑scale data synthesis, cutting‑edge research, and batch training on Hopper/Blackwell GPU clusters.

Nano‑Policy‑DROID – 16 B robot‑specialized model for DROID platform control and strategy generation.

Super‑Text2Image – 64 B model dedicated to high‑quality physical image generation.

Super‑Image2Video – 64 B model for long‑sequence physical video synthesis.

The 16 B Nano version is recommended for individual developers and SMEs, while the 64 B Super version targets labs and large enterprises.

Open‑source ecosystem

Nvidia releases the full code, weights, and six high‑quality synthetic datasets generated with the Isaac Sim physics engine, providing far better physical consistency than web‑scraped data.

Physical‑Interaction‑Scenes (general physical interaction data)

Embodied‑Robot‑Scenes (multi‑category robot operation data)

Autonomous‑Driving‑Scenarios (full‑scene driving simulation)

Warehouse‑Operations‑Scenes (warehouse safety and operation data)

Additional datasets for motion and audio‑video interaction to reduce real‑world data collection cost.

Supporting tools include post‑training scripts for fine‑tuning, an Agent Skills toolkit for environment setup and prompting, and a pre‑compiled Cosmos 3 NIM container with BF16/FP8/NVFP4 quantization (NVFP4 yields up to 2× speed‑up).

Deployment guides

Path 1 – Research / Prototype (Diffusers + Transformers, for individual developers)

conda create -n cosmos3 python=3.11 -y
conda activate cosmos3
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130
pip install transformers accelerate opencv-python
pip install "diffusers @ git+https://github.com/huggingface/diffusers.git"
pip install av cosmos_guardrail

Authenticate with Hugging Face, pull the model weights, and run the inference script.

Path 2 – Production (vLLM‑Omni, enterprise online service)

vllm serve nvidia/Cosmos3-Super \
  --omni --host 0.0.0.0 --port 8000 \
  --cfg-parallel-size 2 --ulysses-degree 4 \
  --use-hsdp --hsdp-shard-size 8

Enable Efficient Video Sampling (EVS) to reduce video token count and significantly speed up inference on modest GPUs.

Practical pitfalls (official bug fixes)

System limitation: only Linux is supported; Windows requires WSL2 or a cloud server.

Memory issue: RTX 5090 running the Nano version needs FP8 quantization to avoid out‑of‑memory errors.

Startup timeout: Super version cluster deployment should increase --init-timeout 1800 to prevent model loading interruptions.

Audio anomaly: the av library must be installed for proper 48 kHz stereo generation.