From One‑Shot Answers to Action Chains: S‑Agent Advances Spatial Intelligence

S‑Agent, proposed by Nanyang Technological University’s S‑Lab together with Ropedia, reframes spatial understanding from a one‑shot answer to an executable action chain: a vision‑language model (VLM) reads the question and plans the next step, dedicated models such as DA3 handle depth, pose and 3D alignment, and a spatial expert translates geometric output into usable evidence.

The opening demo video poses a dynamic spatial reasoning problem—"Standing beside the headphones, back to the entrance, which direction is the camera?"—requiring the model to locate entities across 64 video frames, lift 2D clues into a shared 3D scene, establish a self‑centered coordinate system using the headphones as origin and the "back to entrance" orientation, and finally answer "back‑right". The full action chain consists of three planning rounds and six tool calls, each step constituting part of the reasoning process.

The system architecture is organized into three tool layers. Level 1 gathers 2D visual evidence—selecting key frames, locating entities, and filling missing targets with open‑vocabulary detection. Level 2 lifts these clues to 3D by estimating depth, camera pose, and constructing a bird’s‑eye view to align observations into a common coordinate frame. Level 3 employs spatial experts (counting, measuring, relative positioning, visual orientation, object viewpoint) to convert dense, possibly noisy geometric outputs into high‑level spatial facts usable by the planner. A double‑memory mechanism stores evidence across steps, allowing later actions to build on earlier results.

Ablation experiments show that merely feeding raw 3D outputs to the planner is insufficient; filtering through Level 3 spatial experts to produce task‑relevant measurements and orientation judgments yields the most significant performance gap. Thus, the key is not simply adding more tools but ensuring each action generates verifiable evidence.

In zero‑shot evaluation, S‑Agent achieves 46.4% average on MMSI‑Bench, surpassing Gemini 3 Pro (45.2%) and GPT‑5.4 (41.9%). It improves the camera‑motion subtask by 31.1 points and reaches 60.0% on ViewSpatial‑Bench, with an 81.1% score on the person‑relative‑direction metric, and 58.8% on ReVSI.

Using the S‑300K dataset—approximately 292 k high‑quality action trajectories generated by a strong agent—the authors fine‑tuned Qwen3‑VL‑8B. The resulting S‑Agent‑8B improves from 31.1% to 41.6% on MMSI‑Bench and from 42.2% to 46.8% on ViewSpatial‑Bench, matching or exceeding GPT‑5.4 while using a much smaller model.

The approach generalizes to robot navigation and manipulation, AR/VR spatial assistants, long‑video question answering, and autonomous‑driving perception, all of which require not just visual detection but active selection of the next observation, measurement, and verification, while continuously maintaining object positions, orientations, and changes.

Overall, S‑Agent demonstrates that advancing spatial intelligence is less about scaling model parameters and more about constructing systems that can plan actions, invoke reliable tools, preserve evidence, and distill successful action chains back into compact models.