Why Real2Sim Outperforms Video‑Driven SimFoundry: Building Worlds Directly from Real Space

At the 2026 World Artificial Intelligence Conference, the rapid industrialization of embodied intelligence was highlighted, with many real‑world robot demos and accelerating technology iteration. While embodied large models and control algorithms evolve quickly, diverse Real2Sim simulation routes are also revealing distinct adaptation characteristics and technical boundaries.

In early July, Fei‑Fei Li’s team, together with NVIDIA GEAR and Georgia Tech, launched SimFoundry, a Real2Sim system that builds interactive physical simulation scenes from a single ordinary RGB video, quickly generating many "digital variants" for robot training. This validates the industrial value of Real2Sim.

Unlike the conventional approach of "guessing" the world from 2‑D video, the company 如视 proposes a different path: constructing the world directly from real space. Leveraging a repository of over 60 million real‑world 3D spaces, it has created a reusable, high‑fidelity SimReady "data mine".

Data‑Source Innovation: Real‑World 3D Data Reconstructs Simulation Foundations

Compared with mainstream 2‑D image generation and algorithm‑driven simulation pipelines, 如视 iterates from the data‑capture source, discarding spatial inference and parameter estimation. Its native 3‑D structured data provides three core advantages:

Millimeter‑level precision : Proprietary lidar devices replicate real spaces 1:1, preserving exact dimensions such as tabletop height and door‑frame width, eliminating scale ambiguity.

Zero cumulative error : In large‑scale, long‑distance batch capture and reconstruction, the 3‑D data maintains geometric consistency, avoiding trajectory drift and the need for repeated algorithmic compensation, thus preventing scene distortion.

Full‑element multi‑dimensional information : Each dataset includes depth, normals, material attributes, and precise object poses, all tightly aligned, offering robots a complete, realistic scene description for perception, recognition, interaction, and training.

From Real Space to Trainable Simulation Data

The precision and completeness of real‑world 3‑D data form the low‑level barrier for simulation capability. Using the 60 million‑plus coverage of residential, commercial, industrial, warehouse, and outdoor scenes, 如视 has built a standardized Real2Sim end‑to‑end pipeline that closes the loop of capture, reconstruction, semantic understanding, and scene expansion, efficiently converting physical spaces into AI‑trainable simulation environments.

1. Real‑space capture enriches missing 2‑D dimensions

Instead of relying on lightweight RGB video or planar photos, the system synchronously captures panoramic RGB, depth, lidar point clouds, precise poses, and mesh data with custom equipment. This multimodal acquisition records structure, scale, location, and fine details, providing a solid data foundation for high‑precision 3‑D reconstruction and realistic simulation.

2. Industry‑leading millimeter‑scale 3‑D reconstruction produces standardized digital twins

Multimodal raw data are processed by algorithms to regularize scene structure, refine textures, and calibrate topology and camera poses, outputting complete, orderly digital twin scenes. The workflow refines the original real space rather than generating it through algorithmic inference, enabling downstream semantic understanding and scene editing.

3. Structured 3‑D semantic understanding enables affordance perception

On top of accurate reconstruction, the pipeline performs object segmentation, semantic labeling, and instance recognition, achieving object affordance determination (e.g., load‑bearing, pullable, walkable). Because the data stem from real 3‑D capture, they avoid the hallucination and bias of 2‑D image inference, delivering more precise and reliable semantics—effectively a "scene manual" that robots can read.

4. Unlimited expansion of real‑base scenes meets generalization needs

Leveraging the semantic and physical attributes of real scenes, the system supports object replacement, layout adjustment, and physical parameter modification, while adding realistic material, collision, and lighting effects. From a single real scene, countless variant scenes are generated, all adhering to true physical rules without algorithmic guesswork, enabling low‑cost, high‑efficiency large‑scale simulation for embodied model generalization.

From a technology‑iteration perspective, video‑driven Real2Sim solutions like SimFoundry offer lightweight, low‑cost, rapid scene generation, suitable for fast prototyping. However, native simulation based on authentic 3‑D space—characterized by massive, precise, and structurally processed data—is the core requirement for moving embodied intelligence from lab demos to industrial‑grade, real‑world deployment. The next competitive frontier thus shifts from algorithmic inference to the volume, accuracy, and structured processing of real‑world scene data.