Why Standard Vision‑Language Models + Scale Data Beat Specialized 3D Vision Designs (VLM³)

Meta researcher Zhipeng Cai introduces VLM³, a vision‑language model that challenges the prevailing belief that 3D vision requires task‑specific network designs, loss functions, and data‑augmentation pipelines. The central question posed is whether a standard VLM can replace expert 3D models, and VLM³ provides a negative answer to the contrary.

The authors conduct extensive experiments showing that only two preprocessing steps—camera‑focal‑length normalization and pixel‑space normalization—are sufficient for a vanilla VLM (e.g., Qwen3‑vl‑4B) to learn a wide range of 3D tasks. No architectural changes, marker rendering, or specialized regression formulations are introduced.

Empirical results reveal that VLM³ matches or exceeds state‑of‑the‑art expert models on four major 3D benchmarks:

Monocular depth estimation: accuracy improves from 84 % (DepthLM) to 90 %, matching UniDepthV2 and MoGe2.

Object‑level 3D understanding: surpasses SpatialRGPT while using half the parameters (4 B vs 8 B) and no extra encoder.

Pixel‑matching tasks: outperforms DKM and RoMa.

Camera pose estimation: matches DA3 and exceeds VGGT.

These gains are achieved without the complex designs typical of expert vision models, confirming that a simple, generalist VLM architecture combined with scale data constitutes the most effective paradigm for 3D vision.

The paper (arXiv:2605.30561) and accompanying code (https://github.com/facebookresearch/VLM3) suggest that 3D vision can be unified with other multimodal tasks under a single training framework, simplifying model construction and opening avenues for applications in robotics, autonomous driving, and augmented reality.