VEGA-3D: Unleashing Implicit 3D Priors in Video Generation for Scene Understanding

Problem Statement

The authors ask whether video generation models truly understand the world, i.e., whether they can be leveraged for 3D scene understanding and embodied interaction.

Motivation

Traditional 3D understanding relies on explicit 3D data (point clouds, geometry modules) and costly annotations. Observing recent video diffusion models, the authors notice that generating coherent videos with viewpoint changes forces the model to implicitly learn depth, occlusion, and physical distance. If the model lacked such 3D reasoning, generated frames would collapse into incoherent pixels.

VEGA-3D Framework

VEGA-3D treats a frozen video diffusion model (e.g., Wan2.1) as a "latent world simulator." During the denoising stage, a controlled amount of noise is injected, and the intermediate features from a specific DiT layer (e.g., layer 20) are extracted. These features capture pure 3D structural priors while preserving texture information.

To combine the extracted spatial tokens with the original semantic tokens, VEGA-3D introduces a token‑level adaptive gating mechanism. For each token, a learned weight balances the contribution of semantic priors (answering "what") and generative spatial priors (answering "where"). This avoids the conflict that arises from naïvely adding the two feature streams.

Multi‑view Consistency Analysis

The authors argue that a model’s ability to maintain geometric consistency across viewpoints is a key indicator of true 3D understanding. They measure a multi‑view consistency score and find a strong positive correlation with downstream 3D task performance (normalized overall score, NOS).

Baseline discriminative models such as DINOv3‑Large and V‑JEPA v2 achieve consistency scores of 61.90 % and 72.00 % respectively, while a dedicated 3D extractor (VGGT) reaches only 77.21 %.

In contrast, the video diffusion model Wan2.1 exhibits dramatically higher scores: Wan2.1‑VACE scores 97.04 % and Wan2.1‑T2V scores 96.88 %, demonstrating that the generative training forces the model to build robust 3D object representations.

Experimental Results

Using the adaptive gating and latent‑world simulation, VEGA-3D improves performance on several 3D benchmarks without any extra 3D annotations.

3D Scene Understanding : On ScanRefer, [email protected] rises from 51.7 % to 56.2 %.

Spatial Reasoning : On the VSI‑Bench suite, Qwen2.5VL‑7B with VEGA‑3D shows large gains in relative distance, direction, and path‑planning sub‑tasks.

Embodied AI : Injecting the generative prior into OpenVLA raises the success rate on the LIBERO robot simulation benchmark to 97.3 % for complex object interaction and long‑horizon tasks.

Conclusion and Outlook

VEGA‑3D demonstrates that large video generation models already encode rich physical priors, and that unlocking these priors can dramatically boost 3D scene understanding and embodied AI tasks. The authors suggest that the next breakthrough in 3D spatial reasoning may come not from more annotated 3D data, but from better methods to extract and reuse the dormant physical knowledge inside generative foundation models. As future video models (e.g., Sora, next‑generation Wan) evolve, the potential of VEGA‑3D is expected to grow without bound.