How QuatRoPE Breaks 3D Spatial Reasoning Bottlenecks for Large Language Models

Background and Motivation

Spatial reasoning is essential for embodied agents and underpins 3D visual‑language tasks such as 3D visual grounding and 3D VQA. Existing methods that inject 3D scene representations into large language models (LLMs) suffer from two core bottlenecks: (1) absolute position embeddings lack inherent semantics and early fusion of raw coordinates hampers extraction of relative spatial relations; (2) explicit pairwise relation encoding scales quadratically with the number of objects, quickly exceeding LLM input limits (e.g., InteriorGS scenes contain > 554 objects, leading to ~460 k tokens when using naïve triplet encoding).

Proposed Solution

The authors propose a complete solution comprising three components:

QuatRoPE : a quaternion‑rotation position embedding. It treats each object token as a pure quaternion derived from its 3D coordinates, rotates query and key vectors, and computes attention scores that depend only on relative positions. This yields O(n) tokens that implicitly encode O(n²) pairwise relations, preserving scalability while avoiding pruning errors.

IGRE (Isolation Gated RoPE) : an extension that isolates the new QuatRoPE from the LLM’s native language RoPE. It adds dedicated dimensions for object tokens (zero‑filled for non‑object tokens) and applies a gate so that only object‑object interactions adjust attention scores, preventing interference with textual position encoding.

ASR benchmark : a “Attribute‑Free Spatial Reasoning” test set built from ScanQA. The construction pipeline filters for uniquely answered questions that query object names, removes any attribute descriptors (color, shape, etc.), and reformats the remaining queries into a 3‑choice visual‑grounding format, thereby isolating pure spatial reasoning ability.

QuatRoPE Design Details

QuatRoPE splits each token’s vector into three 3‑dimensional slices, encodes the full 3D coordinate as a single quaternion, and applies a rotation formula that makes the dot product of rotated vectors depend solely on the objects’ relative positions. Unlike axis‑wise encodings such as M‑RoPE, this holistic encoding eliminates the “false‑neighbor” problem where small differences on a single axis inflate attention scores.

IGRE Mechanism

IGRE achieves isolation through two designs:

Dimensional isolation : object tokens receive extra QuatRoPE‑specific dimensions; non‑object tokens are zero‑padded, ensuring that quaternion rotations affect only object‑related subspaces.

Gated adjustment : attention scores are modified only when both interacting tokens are objects; otherwise, the zero‑padded dimensions keep the score unchanged, preserving the LLM’s original language understanding.

ASR Benchmark Construction

The benchmark creation follows three steps:

Sample selection : choose uniquely answered 3D VQA questions from ScanQA that ask for object names.

Attribute filtering : discard any question containing object attributes (category, color, shape).

Format conversion : transform the filtered questions into a 3D visual‑grounding format, reducing the impact of language generation differences across models.

Experimental Validation

Experiments were conducted on baseline models Chat‑Scene and 3DGraphLLM across standard 3D VL datasets (ScanRefer, Multi3DRef, SQA3D) and the new ASR benchmark. Key findings include:

Applying QuatRoPE+IGRE to Chat‑Scene‑1B raised ScanRefer [email protected] from 50.7 % to 55.4 % and Multi3DRef [email protected] from 53.3 % to 58.1 %.

Zero‑shot evaluation on ASR showed [email protected] improvements of 19.48 % for Chat‑Scene‑1B (22.92 % → 27.38 %) and 14.94 % for 3DGraphLLM‑1B (25.89 % → 29.76 %).

Analysis of the “false‑neighbor” issue on ScanRefer demonstrated larger gains when the coordinate‑difference threshold δ is small, confirming that holistic vector encoding mitigates this problem.

Qualitative Results

Qualitative cases on ScanRefer show that QuatRoPE enables the model to correctly identify the nearest object described by relational language (e.g., “the door to the right of the machine”), whereas baseline models often select the wrong instance.

Conclusion

QuatRoPE provides an efficient, scalable 3D position embedding for large models by converting absolute coordinates into relative spatial relations via quaternion rotation, while IGRE isolates this new embedding from the model’s native language RoPE. The ASR benchmark offers a rigorous, attribute‑free evaluation of pure 3D spatial reasoning. Across multiple 3D VL benchmarks and the ASR test, QuatRoPE+IGRE yields substantial performance gains, establishing a new foundation for 3D large‑model reasoning.