S-Space: Inside Multimodal Models' Internal 3D Map and Its Reasoning Failures

Discovery of S-Space in Multimodal Models

Recent demos of GPT-6 Astra show it can build furnished 3D houses in Blender and import them into Unreal Engine 5 for interactive walkthroughs. This raises a question: how do models internally represent spatial relationships between objects? The MirroS team analyzed intermediate activations of several multimodal models and found a stable low-dimensional representation space that encodes left–right, up–down, and near–far positions of objects. They call this space S-Space (Spatial Workspace) — a shared representation that binds objects to locations and can be read, written, and manipulated by perception, reasoning, and output processes.

Extracting and Validating S-Space

Researchers hypothesized that object-position information is linearly encoded in the hidden states of object tokens. Using natural scene images, they constructed spatial queries and used answer gradients to identify three orthogonal directions spanning a continuous linear subspace — S-Space. Reading along these axes yields continuous coordinates for each object on the horizontal, vertical, and depth axes.

Experiments show that coordinates read from S-Space correlate significantly with ground-truth continuous positions. Bypassing the model's language reasoning entirely — simply comparing coordinate differences in S-Space — enables direct spatial question answering on CV-Bench, EmbSpatial-Bench, and SpatialTunnel.

S-Space's three coordinate axes and coordinate readout method

Crucially, S-Space generalizes to unseen data distributions and persists across prompt variations (e.g., asking about color still yields spatial coordinates from the same axes). This indicates S-Space is a persistent internal structure, not a transient representation induced by specific spatial queries. Moreover, though extracted from single-view samples, S-Space forms meaningful coordinates in multi-view inputs: when the prompt designates "View 1", swapping which image is called View 1 causes the readout coordinates to switch to the corresponding perspective, showing that S-Space can be dynamically anchored to a specified viewpoint by language .

Causal Role of S-Space in Spatial Judgments

Reading spatial information from internal representations does not prove the model uses it for judgments; S-Space could be a mere "scoreboard" that records but does not participate. To test causality, researchers directly intervened on object-token coordinates in S-Space — swapping two objects' positions without changing the image or question — and let the model continue computation. This manipulation changed the model's spatial language outputs (left became right, up became down, near became far) while leaving unrelated judgments like color largely unaffected .

After intervening on S-Space, the model's spatial judgments change accordingly.

This demonstrates that S-Space is not just a decodable byproduct but causally participates in the model's spatial reasoning — a genuine spatial workspace that can be both read and written.

Coordinate System Confusion: Cardinal vs. Relative Directions

S-Space likely emerges from pretraining pressure to predict spatial language ("left", "right"). This binds visual entities to relative positions and extends beyond the image frame (e.g., inferring an off-screen ball's location from a goalkeeper's dive direction). However, language brings its own ambiguities and reference frames. Abstract metaphors like political "left"/"right" also align along S-Space's horizontal axis, causing physical visual space and metaphorical linguistic space to share representational structure .

S-Space locates an unseen ball based on the goalkeeper's dive direction.

A concrete failure mode: in Qwen's S-Space, cardinal directions (east, west, north, south) align with image-relative directions (right, left, up, down), and northeast combines rightward and upward components. This matches common map conventions but becomes a trap when a task uses a different reference frame. On MMSI-Bench, a question defined "north" explicitly, yet the model mapped image "right" to "east" and answered "northeast" instead of "north" — it applied the map-like coordinate system without checking the task's reference frame. S-Space makes this mechanism visible: cardinal and image-relative directions share spatial features, so image-direction cues can be mistaken for cardinal directions . Correct spatial reasoning requires first disambiguating the reference frame; the model may read positions correctly but still err by using the wrong coordinate system.

Cardinal and image-relative directions couple in S-Space.

Perspective Rotation Failures and Chain-of-Thought

Beyond reference-frame confusion, models struggle to transform known object coordinates to a new viewpoint. On SpinBench's perspective-taking tasks (imagine viewing a scene from left, right, or back), Qwen3.6-27B achieved 60.96% accuracy with direct answering. Adding chain-of-thought prompting raised accuracy to 83.56%. Tracking recurring object tokens during reasoning shows their S-Space coordinates gradually rotating toward the target perspective: early steps reflect the original view; later steps align with the new view. This confirms chain-of-thought does progressively rotate object positions in S-Space, but the rotation remains inaccurate and unreliable .

External Rotation Outperforms Internal Reasoning

To isolate whether errors stem from initial perception or subsequent coordinate transformation, researchers explicitly decoupled spatial perception from spatial computation: they read object coordinates from S-Space, applied a fixed rotation matrix externally, and answered by deterministic rule. Using the model's own spatial representations but offloading the transformation to an external program, Qwen3.6-27B reached 95.21% accuracy — surpassing chain-of-thought reasoning .

Accuracy comparison: direct answer, chain-of-thought, and read-then-rotate S-Space.

This gap reveals a critical distinction: the model already possesses spatial representations sufficient for much higher accuracy, but stably transforming those coordinates remains a weak link .

Implications: Representation vs. Computation

This work provides a finer-grained diagnostic than final accuracy: separating "does the model have a correct representation?" from "can the model reliably use that representation?" The former may require richer visual and embodied experience; the latter may demand explicit coordinate transformation, simulation, memory, planning, or tool-use mechanisms.

From Anthropic's J-space (a language workspace) to S-Space (a spatial workspace), interpretability is emerging not just as a post-hoc explanation tool but as a scientific instrument for observing and designing intelligence . It helps us see which capabilities emerge naturally with scaling and which still need new training signals, computational mechanisms, or physical experience. For AI heading into the physical world, scaling remains important, but the pivotal question becomes: what world has the model reconstructed internally, and what does it need to reliably transform, simulate, and act within that world? Understanding space is not just seeing it — it is building it inside and learning to change it.

References

https://mirros.ai/blog/s-space

(Interactive Blog) https://mirros.ai/report/s-space.pdf (Technical Report) https://github.com/MirroS-Lab/S-Space (Code Repository)