How to Maximize Cosmos‑3 Training Throughput Without NVLink Using AI Infra Optimizations

Background

Cosmos‑3 is a full‑modal world model that unifies language, image, video, audio, and action‑sequence modeling. The Cosmos‑3‑Nano‑Policy‑DROID variant targets embodied‑intelligence robotics and has become a benchmark model for downstream tasks.

Challenge

The official training setup uses 1,024 NVIDIA GB200 GPUs linked by NVLink and HPN to achieve high scalability. Domestic enterprises and research labs lack such ultra‑high‑speed interconnects, making it essential to extract maximum performance from "general" AI compute platforms.

Optimization Practices

1. Task‑Startup Optimization

The community code caused an out‑of‑memory (OOM) kill after 37 minutes with a peak RAM usage of 1,734 GB. By profiling the data pipeline, redundant fields in ActionBaseDataset were removed and Parquet column‑pruning was applied, reducing peak RAM to 46 GB and cutting startup time to 25 seconds – an 89× acceleration.

2. I/O Throughput Bottleneck

GPU idle gaps were traced to an imbalance where max_samples_per_batch exceeded prefetch_capacity. Increasing DataLoader workers caused CPU contention; profiling revealed that ColorJitter consumed 78.5% of CPU time. Moving this augmentation to the GPU reduced its cost from 1.68 s to 0.08 s, lowering per‑sample processing from 2.12 s to 0.52 s and raising throughput from 1.35 to 2.03 samples / s / GPU (≈50% gain).

3. torch.compile Adaptation

torch.compile fuses kernels to cut launch overhead. The default fusion exceeded the shared‑memory limit on the domestic GPU, causing compile failures. Disabling the mix‑order reduction strategy unlocked a 28.6% throughput increase, raising per‑GPU throughput from 2.03 to 2.61 samples / s.

4. Hierarchical Activation Checkpointing

Full activation checkpointing (AC) on all 36 Transformer layers saves memory but adds recomputation cost. By selectively disabling AC on layers where memory was not a bottleneck, the team converted idle memory into higher batch sizes, improving throughput from 2.61 to 2.69 samples / s / GPU (+3.1%).

5. Multi‑Node Cluster Scaling

Using Baidu Baige’s Elastic RDMA (ERI) network and HSDP parallelism, a 12‑node cluster (96 GPUs) achieved 98.3% scaling efficiency, delivering 251.29 global samples / s compared with the linear baseline.

6. Accuracy Verification

All optimizations are mathematically equivalent or pipeline‑preserving, so training loss curves of the optimized version (blue) match the baseline (red) exactly, confirming loss‑free acceleration.

7. Compute Density

After the full stack of optimizations, the MFU (Model FLOPs Utilization) on the domestic GPU instance reached 0.42, surpassing the official GB200 baseline of 0.23‑0.30.

Key Results

Startup time: 37 min → 25 s (89× faster)

Peak RAM: 1,734 GB → 46 GB (‑97%)

Single‑node throughput: +99.3% (baseline → optimized)

GPU‑side throughput: +50% after I/O fix, +28.6% after torch.compile, +3.1% after hierarchical AC

Scaling efficiency on 12 nodes: 98.3%

MFU: 0.42 (exceeds GB200 benchmark)

Conclusion

Large‑scale embodied‑intelligence deployment requires not only cutting‑edge models but also rigorous AI Infra engineering. The presented case study demonstrates that, even without flagship interconnects like NVLink, systematic optimizations can unlock near‑flagship performance on domestic GPU fleets, reducing training cost while preserving model accuracy.