How OVOW Turns Monocular Video into Physically Simulatable 4D Meshes

Problem & Motivation

Robots operating outside the laboratory require training data that captures real‑world scale, occlusion, contact and motion; otherwise learned policies cannot be transferred. This motivates the Real2Sim problem: converting ordinary video into interactive, physically simulatable scenes at a lower cost than manual modeling.

Challenge

Existing 4D reconstruction methods output implicit fields, Gaussian primitives or point clouds that are suitable for novel‑view rendering but lack closed meshes, accurate scale and temporally stable motion, making them unsuitable for physics engines.

OVOW (One Video, One World) Pipeline

Starting from a single monocular video, OVOW performs the following stages:

Detect object instances and classify their motion type.

Generate cross‑frame masks and complete occluded regions.

Reconstruct static, rigid and deformable meshes for each instance.

Estimate true scale and per‑frame 6‑DoF poses.

Correct ground, gravity, suspension and support relationships.

The result is a 4D mesh world that can be directly imported into a physics engine.

Key Design

No predefined skeletons or category templates are required. A unified vertex‑deformation representation handles static, rigid and non‑rigid motion. Instance meshes serve as the foundation, separating global trajectories from local deformations.

Examples

Desktop rigid objects: A video of a black off‑road vehicle, a dumbbell and a flowerpot is reconstructed into separate rigid meshes that can be moved independently.

Indoor multi‑object scene: Multiple objects on a table are recovered with correct geometry and layout.

Flying eagle deformation: A non‑rigid animal is captured as a topologically consistent deformable mesh.

Airport multi‑rigid: Aircraft, shuttle and terminal are each reconstructed as rigid meshes.

Polar bear deformation: A complex animal with surrounding objects is turned into a deformable 4D mesh.

Simulation & Editing

The reconstructed meshes can be repositioned or subjected to external forces; the physics engine recomputes collisions, tipping and displacement. Three toy‑car tabletop scenarios demonstrate (1) the original layout, (2) an altered layout, and (3) a recombined layout, each producing distinct interaction trajectories rather than a simple video replay.

Benchmark Results

A static and dynamic benchmark of 120 scenes (3–5 objects per scene) provides ground‑truth meshes, trajectories, cameras, depth and segmentation. On the dynamic benchmark OVOW achieves:

Scene‑IoU‑OBB = 0.440

Object‑IoU = 0.210

Processing speed = 3.35 s per frame

Motion‑type recognition = 95.4 %

Pose recovery = 92.4 %

Scene‑gravity stability = 86.8 %

Overall simulation stability = 82.7 %

Speed is one to two orders of magnitude faster than per‑frame methods.

Implications for Physical AI

RGB pixels become an entry point; policies require object boundaries, geometry, scale, trajectories and contact states. OVOW supplies instance‑level 4D meshes that can be edited and simulated to generate diverse counterfactual interaction trajectories from a single observation.

Resources

Paper: "One Video, One World: Turning Monocular Video into Physical 4D Scenes" (arXiv:2606.31388)

Project page: https://onevideooneworld.github.io/

Code repository: https://github.com/SparcAI-Inc/OVOW

Code example

来源：机器之心
本文
约2200字
，建议阅读
5
分钟
本文介绍了 OVOW，解析单目视频生成可仿真 4D Mesh 场景的技术方案。