Let Generative Models Draw Spatial Answers, Not Text Coordinates – Zhejiang’s Agentic Evaluation Framework

Current spatial cognition benchmarks force AI to express answers as coordinates or text, which is counter‑intuitive for visual tasks; humans would simply point or draw.

The Zhejiang University OmniAI team proposes ProVisE (Protocolized Visual Evaluation), a framework that replaces textual or coordinate responses with visual answers. Each visual protocol includes a guidance prompt for the image‑generation model and a parser that extracts colors, positions, or masks from the generated image back into structured predictions. The workflow consists of three steps: the Protocol Router selects an appropriate visual protocol, the model generates a parsable visual answer, and the Protocol Parser converts the pixel output into the benchmark’s required labels, points, masks, or trajectories.

To handle diverse spatial benchmarks, ProVisE incorporates the Agentic Builder, which automatically constructs visual protocols based on task input, answer format, and scoring rules. It operates in three modes—Reuse existing compatible protocols, Build new ones from registered parsing components, and Fallback to an auxiliary VLM when deterministic components cannot cover a task. After construction, each protocol is validated to ensure correct generation, parsing, and metric integration before being frozen for evaluation.

Illustrative examples show the model directly marking a target bus, drawing a robot’s motion path on the scene, producing a depth map to compare distances between points A and B, and adding a second passenger to a cable‑car seat to visualize spatial feasibility. The corresponding parsers then translate these visual answers into conventional coordinate or option predictions.

To systematically assess spatial cognition, the authors built SpatialGen‑Bench, comprising 14 sub‑tasks grouped into four levels: perception (counting, relative depth, orientation, size comparison), understanding (object grounding, relationship analysis, viewpoint judgment, scene modeling), reasoning (multi‑step inference, state prediction, geometric feasibility), and interaction (affordance grounding, navigation, trajectory planning).

Evaluation on SpatialGen‑Bench compared 20 text‑output VLMs and 11 image‑generation models. Claude Fable 5 and GPT‑5.6 Sol scored above 80, while GPT‑5 Image 2 led the generative models with 54.49. Results reveal that generative models excel when tasks can be answered directly with depth maps or spatial markings, reducing information loss from converting continuous space to text. Conversely, VLMs retain an advantage on abstract reasoning tasks such as counting, size comparison, and geometric feasibility, leading the VLM group by 17.6 percentage points on reasoning tasks. The two model families also demonstrate complementarity; for instance, GPT‑5.4 failed on questions that GPT‑5 Image 2 answered correctly 37 % of the time.

Failure analysis shows that 88.03 % of unsuccessful generative model samples did generate an image and a parsable prediction, but the spatial state depicted was incorrect; the primary issue is inaccurate visual representation rather than generation or parsing failures.

In conclusion, the work advocates a new perspective on spatial intelligence where agents express and rehearse spatial states directly in pixel space, while deterministic tools verify the outcomes. This “show‑and‑tell” synergy—generation, verification, and reasoning—may be more promising than merely scaling a single model type toward general spatial intelligence.