Why Machines Need Cognitive Maps to Truly Understand the World: A Survey on Spatial Intelligence

Overview

Recent advances enable AI to recognize images, generate scenes, and plan actions in virtual environments, yet an agent that truly enters a space only sees a limited view and must comprehend the entire world. The authors argue that an agent cannot rely solely on instantaneous observations; it must form a stable, updatable, and inferable internal spatial representation—called a cognitive map —to support downstream reasoning and generation.

Survey Contribution

Researchers from the Chinese Academy of Sciences and several universities released the survey paper Spatial Intelligence from a Cognitive Map Perspective: A Survey . Using the cognitive‑map perspective, they reorganize spatial‑intelligence research and extend the classic biological navigation concept into an internal representation blueprint that connects spatial perception, reasoning, and generation , unifying previously scattered directions.

Cognitive Map Core Properties

The survey defines three essential properties that a cognitive map must possess:

Abstraction : Transform raw sensory inputs (pixels, point clouds, voxels) into structured concepts such as objects, attributes, relations, and topologies.

Globality : Integrate observations from different times, viewpoints, and modalities into a consistent global layout.

Persistency : Maintain and update the internal state over time, recording spatial information and revising it with new observations.

Representation Types

Three families of internal representations are identified:

Metric Representation : Emphasizes geometric structure (coordinates, distances, shapes) using explicit geometry‑based (e.g., 2‑D grids, BEV, point clouds, voxels) or parametric coordinate‑based formats.

Relational Representation : Focuses on topological relations (support, adjacency, reachability) via structured graphs or serialized graph/text formats, facilitating integration with language models.

Hybrid Representation : Combines metric and relational information through hierarchical architectures or feature‑fusion mechanisms, supporting both precise localization and high‑level reasoning.

Spatial Reasoning Paradigms

After a cognitive map is built, the survey categorizes three ways it can be leveraged for reasoning:

Map as Embedding : Encode the map as latent features that participate directly in matching, state propagation, and decision making (e.g., structural state propagation, latent feature matching).

Map as Prompt : Serialize the map into textual, visual, or multimodal prompts for large language or vision‑language models, offering flexibility but incurring information‑compression bottlenecks.

Map as API : Expose the map as an external, queryable interface that can be called, updated, and used to constrain decisions, enabling closed‑loop interaction at the cost of higher system complexity.

Generation from Cognitive Maps

The generation stage reverses perception: it externalizes the internal map into observable space. Two major categories are described:

Static Scene Synthesis : Use map‑derived layout, object semantics, and topological priors to retrieve assets or condition generative models for coherent 3D scene creation.

Dynamic World Simulation : Treat the map as a persistent state that drives temporally consistent simulation of evolving environments.

Application Scenarios

Based on how agents interact with the system, tasks are split into:

Open‑loop Spatial Cognition : Observation, understanding, or generation without real‑time environment modification (e.g., spatial QA, indoor scene synthesis, open‑world generation).

Closed‑loop Spatial Interaction : Continuous perception‑action cycles where the map is constantly queried, updated, and used for embodied navigation or manipulation.

Future Directions

The authors highlight five key challenges for the next generation of spatial‑intelligent systems:

Deeper semantic abstraction beyond object categories to include identity, state, function, and affordance.

Extended global understanding for large‑scale environments, room connectivity, and unobserved region completion.

Long‑term persistence in dynamic settings via 4‑D spatio‑temporal representations and selective forgetting.

Using cognitive maps as generative simulators for future‑state prediction and counterfactual reasoning.

Bridging perception and action by letting map uncertainty and predictions actively influence decision making.

Conclusion

By framing spatial intelligence around cognitive maps, the survey shows how disparate research threads can be compared under a unified mechanism: agents must construct, maintain, and exploit an internal spatial representation that is abstract, global, and persistent to achieve human‑level or superhuman spatial understanding.