VLMs Encode 3D Scene Topology in Latent Space: NeurIPS 2026 Spotlight Findings

Introduction

Humans integrate changing first-person views into stable cognitive maps preserving 3D spatial relations such as adjacency and relative position. This study investigates whether vision-language models (VLMs) form similar internal representations beyond 2D image grids. The paper, Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models by Haoming Wang and Wei Gao (University of Pittsburgh), was accepted as a spotlight at NeurIPS 2026.

Methodology: Three-Step Latent Space Analysis

Step 1: Extract Object-Level Representations from Visual Tokens

Each object corresponds to multiple image patches across video frames. Using synthetic scene object masks, the researchers identify which visual tokens belong to the same object, aggregate them by coverage-weighted pooling, and average across time. This yields a comparable activation vector per object per scene, stacked into a matrix H (Equations 1 and 2). This shifts analysis from scattered 2D patches to object-level relations.

Step 2: Separate Identity Attributes from Spatial Structure via Cross-Scene Averaging and SVD

Placing the same object (e.g., a red cube) in many random scenes varies its position while keeping color and shape constant. Averaging activations across scenes cancels the varying spatial component, estimating stable identity attributes (color, shape). Under approximate linear decomposition, random layout, and zero-mean spatial component assumptions, this average isolates identity directions. Singular Value Decomposition (SVD) then identifies the principal directions of these attributes in latent space. The original representations are projected onto the orthogonal complement of these directions, retaining only position-related variation.

Step 3: Visualize High-Dimensional Spatial Signals with PCA

Principal Component Analysis (PCA) projects the separated object representations onto the top three principal components. Each point represents an object in a scene; the point cloud's arrangement reflects the model's internal relational structure. Side-by-side comparison with ground-truth layouts reveals which layers preserve scene geometry. This is essentially SVD on the separated representation matrix, taking the first three columns of V to obtain a 3D embedding Z.

Key Findings

Middle Layers Exhibit Clearer 3D Scene Structure

Visualizations across early, middle, and late layers show that middle-layer representations most faithfully correspond to true object layouts (ring, grid, random). Early layers emphasize visual features; middle layers highlight spatial organization; late layers shift toward language output. This suggests a layer-wise division of labor in VLMs.

Dirichlet Energy Quantifies Neighborhood Preservation

Beyond visual similarity, the authors use Dirichlet energy to test whether objects adjacent in 3D space are also close in latent space. A Gaussian kernel graph weights edges by 3D proximity; high representation differences between neighbors increase energy. Comparing against a randomized baseline yields an energy ratio per layer. Results show middle layers have lower energy ratios, and this trend strengthens after removing appearance attributes, confirming extractable scene topology.

Theoretical analysis (Theorem 1) proves that minimizing Dirichlet energy under non-collapse constraints yields optimal representations whose PCA visualization corresponds to Laplacian eigenmaps, mathematically linking neighborhood preservation to interpretable geometric structure.

Intervention Along Spatial Directions Alters Coordinate Readouts

Linear probes trained to predict 3D coordinates from activations were used to identify spatial directions. Perturbations along these directions caused directional changes in probe readings, while control perturbations of equal magnitude but orthogonal to probe directions did not. This provides causal evidence that the extracted directions encode spatial information.

From Representation Understanding to Capability Improvement

Adding Dirichlet energy as a fine-tuning objective with synthetic 3D coordinate supervision, the team performed 500-step LoRA fine-tuning on selected middle layers. On VSI-Bench and MindCube benchmarks, this yielded transfer gains. For example, Qwen2.5-VL-7B accuracy on VSI-Bench simple relative direction tasks rose from 47.9% to 60.0% (a 12.1 percentage-point improvement) over standard LoRA.

This demonstrates that latent space analysis can guide training design, and reinforcing spatially relevant representations enhances VLM spatial understanding and reasoning.

Conclusion

The core contribution is making VLM latent 3D structure observable, comparable, and testable. By disentangling appearance, visualizing layouts, measuring neighborhood relations, and performing targeted interventions, the work transforms the vague question "does the model understand space?" into concrete representational analysis, providing tools for further research on spatial cognition in VLMs.

Resources

Paper: https://arxiv.org/abs/2605.07148 Code:

https://github.com/pittisl/vlm-latent-shaping