From One Video to a Simulatable Dynamic World: OVOW’s 4D Reconstruction Breakthrough

Introduction

Robots deployed outside the lab need training data that captures real‑world scale, occlusion, contact, and motion. The paper One Video, One World: Turning Monocular Video into Physical 4D Scenes (ECCV 2026) proposes OVOW, which turns a single monocular video into an interactive, simulatable 4D world, reducing data‑collection cost compared with manual modeling or specialized sensors.

Challenges of 4D Reconstruction

Existing 4D reconstruction methods output implicit fields, Gaussian primitives, or point clouds suitable for novel‑view rendering but lacking independent objects, closed meshes, true scale, and stable temporal motion, making them unsuitable for physics engines.

OVOW Pipeline

OVOW operates in three stages:

Detect instances and classify motion types, generate cross‑frame masks, and complete occluded regions.

Reconstruct static, rigid, and deformable meshes while estimating real‑world scale and per‑frame 6‑DoF poses.

Correct ground, gravity, and support relationships to produce a coherent 4D mesh world ready for simulation.

From Rendering to Simulation

Outputs editable, collidable instance‑level meshes: rigid objects retain true scale and per‑frame poses; non‑rigid objects preserve topologically consistent temporal deformation. The system also estimates ground and gravity to fix floating, penetration, and incorrect support.

Pipeline Details

The pipeline chains visual foundation models for scene understanding, geometry and motion recovery, and physical assembly without training a dedicated large model. Instance meshes form the base; overall trajectories and local deformations are separated, enabling simultaneous handling of rigid bodies (e.g., vehicles, aircraft) and non‑rigid entities (e.g., birds, bears).

Representative Cases

Desktop Rigid Objects : A video of a black off‑road car, dumbbell, and flowerpot is reconstructed into a 4D mesh where each object can be individually manipulated.

Indoor Multi‑Object Scene : Multiple tabletop objects are recovered with correct geometry and layout.

Deforming Eagle : A wild‑eagle video yields a topologically consistent non‑rigid 4D mesh.

Airport Multi‑Rigid : Aircraft, shuttle buses, and terminal structures are reconstructed as separate rigid meshes.

Polar Bear Deformation : A polar bear and surrounding objects are captured as deformable meshes.

From Reconstruction to Physical Simulation

Because the reconstructed scene encodes physical properties, OVOW can reposition objects or apply external forces, allowing a physics engine to compute new collisions, tilts, and displacements. Three tabletop simulations demonstrate different interaction outcomes solely by changing initial layouts.

Reconstruction Quality and Pipeline Reliability

The authors built static and dynamic benchmarks containing 120 scenes each, with 3–5 objects per scene and ground‑truth mesh, trajectory, camera, depth, and segmentation. On the dynamic benchmark OVOW achieves Scene‑IoU‑OBB = 0.440, Object‑IoU = 0.210, and runs at 3.35 seconds per frame. Motion‑type recognition, pose recovery, scene‑gravity stability, and gravity‑correction rates are 95.4 %, 92.4 %, 86.8 %, and 82.7 % respectively, a speedup of one to two orders of magnitude over frame‑by‑frame methods.

Impact on Physical AI Data Pipelines

OVOW supplies instance‑level 4D meshes that can be fed to simulators to generate diverse counter‑factual interaction trajectories from a single observation. It can serve as the upstream component of D4RT‑style pipelines, providing meshes, cameras, depth, and motion for downstream reconstruction, tracking, and robot‑policy training.

Conclusion

OVOW transforms a single video into a manipulable physical world, enabling repeated interventions such as object repositioning, collision generation, and long‑tail state synthesis, thereby turning a one‑time observation into a rich data source for world models and robot learning.

Paper: https://arxiv.org/abs/2606.31388

Project page: https://onevideooneworld.github.io/

Code: https://github.com/SparcAI-Inc/OVOW