3D Scene Graphs: Open Challenges and Future Directions

Introduction

Autonomous agents operating in unstructured real environments require both precise geometric grounding and high‑level semantic understanding. Traditional SLAM, point clouds, and mesh maps provide accurate geometry but lack reasoning about object categories, relations, and actions. Pure language or implicit representations encode semantics but miss physical grounding. 3D Scene Graphs (3DSG) represent scenes as graphs whose nodes correspond to objects, rooms, floors, or environments and whose edges encode spatial, semantic, hierarchical, action, or temporal relations, together with geometric and multimodal attributes.

Three forces drive the emergence of 3DSG: (1) reliable large‑scale geometric maps from SLAM and 3D reconstruction, (2) improved semantic extraction from object detection, instance segmentation, and foundation models, and (3) the rise of large language and vision models that serve as interfaces between language, perception, and action.

Formal definition

A 3DSG is a graph G = (V, E) where each node v ∈ V is grounded in 3D space and may represent an object, object part, room, floor, or region. Nodes carry geometric attributes (centroid, bounding box, shape) and semantic attributes (category, material, color, multimodal embedding, textual description). Edges e ∈ E can be directed or undirected and may encode spatial adjacency, semantic relations, functional relations, reachability, containment, or action effects.

Modeling extensions

Hierarchical structure – Indoor environments often follow a hierarchy (environment → floor → room → place → object). Outdoor scenes lack clear room boundaries and are organized by roads, intersections, or functional zones. The open challenge is whether hierarchy should be defined by geometric boundaries, task semantics, or topological structure, and how to infer a universally useful abstraction.

Dynamic modeling – Real scenes change over time: objects move, doors open/close, humans act, robots modify the environment. 3DSG must capture temporal changes, data association, uncertainty, and future states. Existing methods mainly track object trajectories and rarely model evolving relations or high‑level dynamic abstractions.

Functional and actionable properties – Objects may have affordances (e.g., a cup is graspable, a door is openable). Encoding affordances and action effects is essential for moving from descriptive representation to task‑driven spatial intelligence.

Construction pipeline

Graph construction extracts structure from RGB‑D, point clouds, meshes, SLAM trajectories, instance segmentation, 2D/3D detection, language descriptions, or multimodal model outputs. The problem is organized along five dimensions.

Processing

Methods are classified as online (incremental updates during robot operation) or offline (batch fusion for high‑quality semantics).

Nodes

The primary decision is what constitutes a node. Common choices are objects, but nodes can also be rooms, places, functional areas, open‑vocabulary entities generated by VLM/LLM, agents, or affordances.

Edges

Edges may represent spatial adjacency, support, containment, relative position, semantic relations, hierarchical parent‑child links, or action relations. Edge inference can rely on geometric rules, learned models, language models, or multimodal reasoning. No unified taxonomy for edge types exists across the literature.

Priors

Task knowledge, knowledge bases, LLM/MLLM outputs, physical constraints, or commonsense (e.g., “cups are usually on tables”) are incorporated as priors. Priors can introduce hallucinations or conflict with geometric observations.

Consistency

Multi‑view, multi‑time, multi‑sensor, and multi‑abstraction consistency must be maintained. Current work focuses on geometric alignment; semantic and relational consistency, as well as long‑term memory handling, receive less attention.

Applications and evaluation

Beyond accurate representation, 3DSG supports downstream tasks. Evaluation is grouped into intrinsic metrics and task‑oriented benchmarks.

Intrinsic evaluation

Metrics include precision, recall, F1, mean IoU, Chamfer distance, 3D reconstruction errors, and relation recall. Ground truth for non‑object nodes and abstract relations is often unavailable, leading to fragmented protocols.

Scene understanding

3DSG enables cross‑modal grounding, language queries, relational reasoning, and multi‑hop question answering (e.g., “where is the window next to the red chair”). Recent pipelines serialize graphs to JSON or textual graph formats for LLM reasoning. When graph size exceeds LLM context windows, retrieval, compression, or sub‑graph selection techniques are required.

Planning and navigation

High‑level language goals are grounded to objects, rooms, or traversable areas and then translated into path or task planning problems. 3DSG can feed symbolic planners (PDDL/LTL) and provide semantic topological maps for navigation. Open challenges include handling dynamic scenes, fidelity of the map, and tighter coupling between task planning and low‑level motion planning.

Manipulation

Structured context from 3DSG supports grasping, interaction, object search, and long‑term tasks. Support relations, occlusion, actionable states, and affordances are critical. Future work aims to combine long‑term memory, verifiable action effects, and vision‑language‑action models.

Emerging applications

Open‑vocabulary spatial QA, embodied long‑term memory, digital twins, interactive scene editing, future object state prediction, and serving as an explicit world model for physical AI push 3DSG beyond traditional robotics maps toward broader spatial‑AI representations.

Open challenges

Lack of unified standards for node and edge definitions, hierarchy, dynamics, and functionality.

Unstable construction pipelines, especially for open‑vocabulary recognition, multi‑view consistency, dynamic objects, long‑term updates, and hallucination control.

Insufficient evidence that 3DSG outperforms simpler representations in complex real‑world tasks.

Fragmented evaluation lacking cross‑task, cross‑dataset, and cross‑hierarchy benchmarks, which hampers systematic comparison.

Future directions

Move from static scene description to task‑driven, dynamically consistent, verifiable, open‑vocabulary, and language‑interactive structured world models. Such representations provide an interpretable, queryable, and composable intermediate layer linking geometry, semantics, language reasoning, and action planning.

Paper link: https://arxiv.org/abs/2606.19383

Project website: https://3dscenegraphs.com