How RynnWorld‑4D Gives Robots a 3D‑Future Vision with 4D World Modeling

Problem background – Recent video‑generation models for embodied intelligence predict future RGB frames but lack depth and explicit motion fields, creating a representation gap that prevents robots from accurately estimating distances, 3D motion, and avoiding physical implausibilities.

Representation design: RGB‑DF – RynnWorld‑4D adopts an RGB‑Depth‑Flow (RGB‑DF) representation instead of RGB‑D‑Normal or NeRF/3D‑Gaussian. Depth provides per‑point 3D positions, while optical flow, when back‑projected with depth, yields 3D scene flow, directly aligning with robot action spaces.

Architecture – The core is a three‑branch Transformer (RGB, depth, flow) built on Wan 2.2‑TI2V‑5B (30‑layer DiT, hidden dim 3072). Each branch has independent self‑attention and FFN; cross‑modal alignment is achieved by inserting Joint Cross‑Modal Attention (JA) modules every three blocks, with frame‑level 3D RoPE ensuring attention only between spatially corresponding tokens. Ablation shows sharing FFNs degrades performance, confirming heterogeneous latent spaces.

Training stages – Stage 1 adapts each modality separately (JA disabled). Stage 2 freezes backbone weights and trains JA modules to establish cross‑modal pathways. Stage 3 fine‑tunes all parameters on the full dataset. A Branch Dropout mechanism randomly drops depth or flow inputs (0.2 in Stage 2, 0.1 in Stage 3) to force JA to reconstruct missing modalities.

Data construction – Rynn4DDataset 1.0 provides 2.54 × 10⁸ frames of RGB‑Depth‑Flow triples, combining first‑person activity videos (Epic‑Kitchens, EgoVid) with diverse robot operation recordings (RoboMIND, RDT‑1B, Galaxea, RoboCoin, AgiBot). Each frame receives language captions (via Qwen3‑VL), depth maps (Depth Anything 3), and dense flow (DPFlow). Removing this large‑scale pre‑training raises flow error (AEPE) from 0.170 to 0.729, highlighting data scale importance.

Policy learning – RynnWorld‑4D‑Policy extracts 4D latent features from the 15th Transformer block at diffusion timestep t=500, compresses them with a Flow Former, and generates 10‑step actions (54‑dim) via a lightweight flow‑matching head and 4‑step ODE sampling. Inference on a single RTX 5090 (FP8 + FlashAttention 3) takes ~1.1 s, yielding ~9 Hz effective control frequency.

Experimental results – Quantitative evaluation on 50 test videos shows RGB quality comparable to Wan‑14B (SSIM 0.754 vs 0.536), depth δ₁ = 0.610 (2.2× TesserAct), and flow AEPE = 0.170 (best among 4D models). Qualitative results demonstrate cross‑modal consistency, physical realism, and temporal stability. In six real‑world dual‑arm tasks (35 trials each) on a Tianji M6 robot, RynnWorld‑4D‑Policy achieves 28.57 % success on Hand‑over (vs 2.86 % for π₀), and 65.71 % on Lid Placement and Bowl Stacking, outperforming baselines by 8–10 percentage points. Ablations confirm that RGB alone drops success by 10–20 pp, depth benefits geometry‑heavy tasks, and flow improves motion‑sensitive tasks.

Outlook – The work moves embodied world models from 2D video imagination toward 4D physical prediction. Future directions include accelerating inference beyond 9 Hz and extending to multi‑view setups for collaborative robots.