3D Vision-Language Models: From Geometry to Open-Vocabulary & Embodied Interaction

1 Introduction

Understanding 3D objects and scenes requires joint processing of geometry, appearance, and semantics. Unlike 2D images which are regular pixel grids, 3D data can be unordered point sets, meshes with topological connectivity, multi-view renderings, sparse voxels, or Gaussian primitives. Different representations preserve different information and require different encoding strategies.

Traditional 3D deep learning operates in closed-set settings: training and testing share fixed categories, mapping inputs to predefined labels. MVCNN and PointNet demonstrated multi-view and point-cloud encoding paths, but their task-specific outputs do not naturally form a cross-modal space queryable by text or images.

CLIP offers an alternative: learning a shared embedding for images and text so that matching content has high similarity. 3D vision-language models further align geometric embeddings with the existing image-text space, enabling natural language to participate in object recognition, open-vocabulary understanding, and cross-modal retrieval.

The paper distinguishes two model families. 3D Vision-Language Models (3D VLM) primarily output embeddings or similarity scores for comparison, ranking, and identification. 3D Vision-Large Language Models (3D VLLM) feed 3D features or tokens into a large language model to generate descriptions, answer questions, respond to instructions, or handle spatial relations. This distinction avoids conceptual conflation: matching text to shape does not guarantee reliable answers to complex spatial questions, and fluent generation does not prove geometric fidelity.

2 Representations and Cross-Modal Alignment

The paper starts from a matching triplet: a 3D object or scene representation S , a related image I , and a related text T . Three encoders extract features, then projection layers map them to same-dimensional, normalized embeddings. Text may be category names, object descriptions, questions, referring expressions, or instructions — not just a single annotation form.

In CLIP-style methods, image and text encoders are often initialized from pre-trained models and can remain frozen; the 3D encoder learns representations compatible with the visual-semantic space. However, which components are trainable depends on the method design; not all 3D models can be summarized as the same freezing scheme.

2.1 3D Inputs and Encoders

Multi-view Images

Rendering the same object from multiple viewpoints, processing each view with a shared 2D encoder, and aggregating into an object-level feature. Advantage: leverages mature image foundation models; cost: viewpoint and rendering choices affect which geometric information is visible, and occluded or unprojected structures may be lost.

Point Clouds

Point clouds are unordered sets of 3D coordinates, optionally with normals or colors, and are a key output of LiDAR and depth sensors. Models must handle permutation invariance, local neighborhoods, and long-range geometric relations. PointNet, dynamic graph networks, point-cloud Transformers, and masked pre-training provide different encoding strategies; sparse convolutions suit sparse spatial data. The paper also mentions research decomposing point clouds into compact geometric primitives.

Meshes

Meshes describe surfaces via vertices, edges, and faces, preserving explicit connectivity — a core representation in computer graphics. MeshCNN focuses on edge-based operations and pooling; MeshNet uses face-level spatial and structural features. The difficulty lies not only in coordinates but also in irregular topology and connectivity.

Implicit Fields

Occupancy functions or signed distance functions express surfaces as continuous functions rather than enumerating all points or faces. Works like Michelangelo align such shape latent representations with images and language. This line shows that a shared semantic space can connect different geometric representations, but does not imply their geometric encodings are identical.

3D Gaussian Splatting

3D Gaussian Splatting (3DGS) describes a scene with anisotropic Gaussian primitives containing center, covariance, opacity, and color, then forms images via differentiable rendering. It carries both geometry and appearance, facilitating connection to 2D visual supervision. Although Gaussian centers are spatial points, the full Gaussian representation includes additional attributes and cannot be simply equated to coordinate-only point clouds.

Representation choice is therefore not irrelevant preprocessing. Object-level retrieval, scene interaction, asset generation, and robot perception have differing requirements for detail, appearance, metric precision, and computational cost.

2.2 Geometric View of Contrastive Alignment

The same chair can appear as a point cloud, a rendered image, or the text "wooden chair", yet the three raw feature spaces are not naturally comparable. Even with equal vector dimensions, distances may reflect encoder differences more than semantic correspondence.

Contrastive learning projects them into a common space, increasing similarity for matching pairs relative to other candidates in the batch. Intuitively, this is a semantic map: different modalities expressing the same content move closer, while mismatched content separates in relative terms.

When embeddings are L2-normalized, they lie on a unit hypersphere where dot product equals cosine similarity. Alignment can thus be viewed angularly: matching 3D, image, and text vectors should have smaller angles. This angle is in embedding space, not the object's real-world orientation.

"Positive" and "negative" samples are defined by the anchor's correspondences: paired image or text is positive, other batch candidates are negative. They are not value judgments on content quality. A shared space improves cross-modal comparison, but semantic proximity alone does not guarantee the model has preserved all geometric details.

2.3 Contrastive Alignment in 3D Vision-Language Models

During training, for a batch of N matched triplets, 3D-image and 3D-text similarity matrices are constructed. The main diagonal represents matches; off-diagonals are other candidates. Similarity is typically the dot product of two normalized vectors divided by a temperature parameter, which controls the concentration of the matching distribution.

Directional InfoNCE loss can be understood as cross-modal N-way classification: given a 3D anchor, identify its paired sample from the entire batch of images or texts. The loss increases the probability of the correct pairing relative to other candidates, not merely pulling two points together in isolation.

Matching is usually bidirectional. For example, 3D-to-image and image-to-3D each have a directional loss, averaged; 3D-to-text uses a similar scheme. The paper gives the trimodal objective as the sum of two bidirectional losses:

Total alignment loss = 3D↔Image loss + 3D↔Text loss.

If image and text encoders inherit from CLIP, the 3D encoder can leverage the existing visual-semantic structure to learn comparability. Main differences across methods remain the 3D representation, backbone, supervision quality, pairing construction, and trainable components — not merely whether the loss form looks the same.

3 A Taxonomy of 3D Vision-Language Models

3.1 From Closed-Set 3D Recognition to Language Alignment

Early models classified within fixed categories; later graph networks and Transformers improved geometric encoding, but task labels still constrained outputs. Language alignment connects the fixed classification head to a continuous semantic space, allowing text-defined categories or object queries at inference.

Zero-shot classification means no dedicated label training for the target categories; it does not imply the model was never pre-trained, has never seen related semantics, or is reliable for any new category. Open-vocabulary expands the query interface, but recognition quality still depends on data coverage, description style, and domain shift.

3.2 Projection-Based 3D Vision-Language Adaptation

Projection methods convert irregular 3D structures into 2D views so that existing image encoders can process them. PointCLIP uses multi-view depth maps compared with text category features; PointCLIP V2 improves projection and 3D-relevant prompting for better transfer; EPCL explores using frozen CLIP features for point-cloud understanding.

The appeal is direct reuse of 2D pre-trained capabilities, sometimes without learning a dedicated 3D encoder. However, they depend on viewpoint selection, rendering quality, and information preserved by projection. Strong 2D semantics do not guarantee equally complete hidden structure, precise scale, or topology.

3.3 Native Joint Embedding Space

Joint embedding methods directly train a 3D encoder so that native geometric inputs produce features compatible with image and text. CG3D and CLIP2Point explored early transfer; ULIP adopts the widely used trimodal alignment form, linking a point-cloud encoder with frozen image and text encoders.

Subsequent work advanced data, supervision, and model scale: OpenShape emphasizes large-scale diverse 3D assets; Uni3D explores scalable backbones; ULIP-2 uses auto-generated captions; others improve augmentation, cross-modal supervision, modality gaps, and lightweight adaptation.

The tutorial also discusses extensions toward universal space encoders. Concerto combines 3D self-distillation with 2D-3D joint embedding, then connects to language space; Utonia studies unified self-supervised encoding across heterogeneous point-cloud domains. These works should not all be summarized as the same triplet training pipeline.

Native encoding typically preserves more geometric information than projection, but requires large-scale paired data, faces weak supervision or noisy descriptions, and incurs high pre-training cost. Geometry preservation is not automatic; it is still determined by the encoder and training objective.

3.4 3D Gaussian Splatting and Vision-Language Models

Gaussian representations place geometry and renderable appearance in the same structure, enabling links between 3D primitives and 2D visual supervision. UniGS extends trimodal alignment to Gaussian features; TIGaussian decomposes latent branches for different attributes and uses multi-view fusion; CLIP-GS focuses on Gaussian tokenization and viewpoint-aware supervision for cross-modal retrieval.

This line benefits attaching semantics to interactive 3D scenes, but still depends on reconstruction quality and may introduce high storage and per-scene processing costs. Object-level Gaussian retrieval and scene-level semantic segmentation/editing share the representation but are not the same task.

3.5 3D Vision-Large Language Models

These models typically consist of a 3D encoder or tokenizer, a learned projector, and a pre-trained language backbone. The projector maps 3D features into the language model's input space; the language model generates responses conditioned on geometric information and text instructions.

PointLLM and ShapeLLM exemplify this common architecture; PointAlign uses feature regularization to preserve geometry; Multi-3DLLM extends relationship modeling to multiple objects; SAGE converts point clouds into discrete tokens; N3D-VLM emphasizes native 3D grounding and spatial reasoning.

Shifting from embedding-based to generative models changes the learning objective: from cross-modal comparison to language generation. This gains interaction capability but introduces hallucination, geometric fidelity, token efficiency, and scarcity of 3D instruction data. Fluent language is not a proxy for spatial correctness.

4 Current Trends and Emerging Applications

This section unfolds in four subsections; the application figure illustrates eight task forms. They share the 3D-language connection capability but have distinct outputs and evaluation goals, and should not be conflated into a single unified task.

4.1 Recognition, Localization, and Retrieval

Zero-shot classification compares 3D objects with text categories, outputting candidate category scores; cross-modal retrieval queries an asset library with text or image, returning ranked 3D objects; visual grounding localizes a referring expression to a spatial region in a scene.

PointCLIP series transfers image-text knowledge via projection; ULIP, OpenShape, and Uni3D support recognition and retrieval via native embeddings; CLIP-GS extends retrieval to Gaussian representations. Multi-object and native geometry models further support grounding and relation understanding.

These three tasks share language alignment as a foundation; the difference lies in output being category, ranking, or position. "Retrieving a chair model" and "finding that specific chair in the current room" differ in granularity and geometric demands.

4.2 Scene Understanding and Spatial Reasoning

3D QA moves from single-object attributes to multi-object relations. PointLLM and ShapeLLM handle object-centric language interaction; Multi-3DLLM extends relation modeling to multiple objects; N3D-VLM stresses native 3D grounding.

Scene-level methods also embed semantic features into Gaussian representations: Language-Embedded Gaussians study cross-view consistency; Gaussian Grouping and Unified-Lift support instance segmentation, grouping, and editing; OpenGaussian provides point-level open-vocabulary understanding; GaussianCut supports interactive object separation; SceneSplat explores predicting language-aligned features without per-scene optimization.

Scene semantic labels, object boundaries, and spatial relations must be jointly accurate. Text queries, clicks, and spatial prompts can form interactive entry points, but if geometric grounding is insufficient, correct category words cannot guarantee correct position, orientation, and relations.

4.3 3D Generation

Generation tasks require synthesizing geometry and appearance, not just comparing existing objects. The paper discusses MeshGPT and MeshFlow as geometric generators, while clarifying they belong to geometry generation and should not be automatically classified as vision-language models merely because they produce 3D content.

TRELLIS generates structured 3D content conditioned on text or image, supporting mesh, radiance field, or Gaussian outputs; SAM 3D studies recovering geometry, texture, and layout from a single image. Geometric generation, multi-modal conditional generation, and image-based reconstruction are complementary directions with different input conditions and supervision targets.

4.4 Embodied AI

Embodied tasks connect semantic and spatial understanding to physical action. CLIPort combines semantic and spatial pathways for manipulation; Gato explores a general sequence model across perception and control; RT-2, Octo, OpenVLA, and GR00T N1 are vision-language-action models mapping observations and instructions to actions.

The authors particularly note that most of these systems primarily rely on 2D observations, so they cannot all be called explicit 3D vision-language models. 3D representations can supplement metric depth, object extents, and spatial relations, but action control still faces latency, scarcity of 3D-language-action data, robustness in unseen environments, and geometric precision maintenance.

"Being able to say how to do it" and "being able to execute it safely and accurately" remain separated. Language interface, geometric perception, and action policy need separate verification; one capability cannot vouch for the others.

5 Conclusion and Future Directions

The tutorial traces paths from 2D projection to native joint embedding, to language-aligned Gaussian representations, and to 3D vision-large language models. The common goal is making language an interface to access 3D objects and scenes; different routes make different trade-offs between reusing 2D knowledge, preserving native geometry, expressing appearance, and supporting generative interaction.

Core challenges summarized include: lack of scalable encoders that generalize across point clouds, meshes, Gaussians, and structured latent representations; limited use of explicit geometry in current embodied models; multimodal alignment efficiency, scale of 3D-language-action data, and transfer from synthetic assets to real sensor data.

These are future research directions, not a unified 3D foundation model already achieved. The paper also does not provide a unified performance comparison of all routes under the same data, budget, and metrics; therefore one cannot claim any representation or method is optimal for all tasks.

For readers, the tutorial provides a key framework: first discern what a 3D representation preserves, then judge whether the task needs similarity matching or language generation, and finally examine whether geometric information is preserved through alignment and interaction. Progress in 3D vision-language learning is not just making models "describe 3D", but establishing verifiable links among semantics, geometry, and actual operation.

Code example

来源：专知
本文
约6000字
，建议阅读
10
分钟
能够把文本与形状匹配，不意味着已经能可靠回答复杂空间问题；能够生成流畅回答，也不证明其描述符合真实几何。