UrbanGround: Testing MLLM Agents in Real-Scale Hong Kong City Navigation

A joint team from Shanghai Jiao Tong University, National University of Singapore, Meituan, Chinese University of Hong Kong, Shanghai University, and Oxford University introduced UrbanGround , a city-scale interactive environment built from Hong Kong's real 3D geographic data to systematically evaluate multimodal large language model (MLLM) agents on three spatial capabilities: Grounding (local spatial anchoring), Persistence (maintaining spatial knowledge over time), and Adaptation (adjusting to environmental changes).

Dataset: Real Hong Kong as an Interactive Sandbox

UrbanGround uses two core data sources from the Hong Kong Lands Department's 3D Digital Map:

3D Visualisation Map : Reconstructed from oblique aerial imagery, delivered as Cesium 3D Tiles in WGS84 coordinates. It provides textured 3D meshes covering the entire territory — buildings, roads, terrain — used for both rendering and physics collision, so structures and elevation changes physically constrain agent movement.

3D Pedestrian Network : A connectivity graph of walkable space (nodes = locations, edges = walkable segments). Agents move freely in continuous space; the network serves as a post-hoc reference to detect violations such as jaywalking or leaving pedestrian zones.

Dynamic simulation adds time of day (day, dusk, night), weather (rain, fog), temporary road closures, and virtual pedestrians from Microsoft Rocketbox moving along the pedestrian network. This enables repeated testing of the same location under varying conditions.

The team created 810 human-verified base task instances across diverse Hong Kong districts, covering different street layouts, topography, and vertical pedestrian connections (bridges, tunnels). Each task was confirmed solvable by a human tester within the same 100-step limit imposed on models.

Agent Framework: Observation–Decision–Action Loop

UrbanGround is an evaluation framework, not a new model. It connects external MLLMs to a Unity simulation via three layers:

Geospatial layer: unifies 3D city model, pedestrian network, and agent trajectories.

Simulation layer: handles continuous motion, physics collisions, and dynamic environmental changes.

Agent layer: bridges the MLLM and Unity. At each step the model receives a first-person RGB image, the task instruction, and the text interaction history. For broader spatial reasoning, the agent can open an interactive map to pan, zoom, and rotate — but the map provides no routing, distance, or navigation API; the model must still infer direction, feasible paths, and self-location.

Action space (first-person): move, accelerate, adjust view, jump, open map. Map mode: select position, pan, zoom, rotate. One action per step; Unity executes it and returns the next observation, forming a closed loop.

Five-Level Evaluation Design

The benchmark progresses across five levels:

Visual recognition, direction judgment, active exploration.

Short-distance navigation.

Long-distance navigation.

Instruction-based navigation, destination inference, multi-waypoint planning.

Dynamic adaptation: time, weather, road closures, moving pedestrians altered during task execution.

Models tested (all via same interface, no UrbanGround-specific fine-tuning): GPT-5.5, GPT-5.4, GPT-5.2, Claude-Opus-5, Claude-Opus-4.6, Gemini-3.6-Flash, Gemini-3.1-Pro, Doubao-Seed-2.0-Pro, GLM-5V-Turbo, Kimi-K3.

Key Findings

Local perception strong, but direction judgment weak

On Level 1 tasks, most models accurately identify nearby buildings, shops, and urban objects. Active exploration to find out-of-view targets shows only modest accuracy drop. However, direction judgment is near random for some models : they recognize a landmark but cannot reliably place it relative to their heading. Recognizing an object ≠ building a reliable spatial bearing.

Action ≠ reasonable movement

Trajectories reveal that correct answers can come with implausible paths. Example: GPT-5.5 found a target bank near Beijing Tongrentang but crossed the road directly, leaving the registered pedestrian network. Models locate goals but lack stable understanding of traversable space and road constraints.

Short-range navigation works; long-range collapses

Success rates drop sharply as route length grows beyond a few city blocks. Long-distance navigation is almost never completed fully by any model. Even within short-distance tasks, success rate declines continuously with interaction distance. Trajectory analysis shows >50% of failed long-range trajectories end closer to the goal than the start ; multi-waypoint tasks reach only ~10–20% of specified waypoints on average. Models execute several correct local steps but cannot sustain global route and progress tracking.

Obstacle handling inconsistent

Some runs show models detecting blocked direct paths and using ramps or footbridges to continue. Others show repeated forward moves toward the goal direction despite being blocked, failing to replan. “Moving toward goal” and “finding a truly passable route” remain distinct challenges.

Environmental changes expose fragile state maintenance

Under reduced visibility (night, rain, fog), local QA accuracy drops, but short-range navigation does not follow a consistent trend — indicating navigation failure depends on route-state maintenance and action execution, not just vision. When roads close mid-task or moving pedestrians appear, models often continue locally plausible actions without rebuilding a valid route; staying in pedestrian zones does not guarantee correct handling of dynamic obstacles.

Conclusion

Current MLLMs possess basic local spatial understanding and short-range action ability, but these capabilities do not naturally extend to reliable city-scale navigation. The core gap lies in long-term spatial state maintenance, error recovery, and dynamic replanning . Advancing spatial intelligence requires not only sharper perception but mechanisms to retain critical information across observations, correct errors, and compose successive local judgments into sustained, reliable action in a changing city.