MapSatisfyBench: From Task Completion to Decision Satisfaction for Map Agents

Introduction

Large language models are turning map dialogue systems from pure instruction‑execution tools into understanding assistants, but existing evaluation paradigms still focus on task completion and factual correctness. Completion alone does not guarantee that the solution is acceptable to users.

What is MapSatisfyBench

MapSatisfyBench, jointly created by Amap’s AI Evaluation Center and Peking University, is the first benchmark that places user satisfaction at the core of map‑agent evaluation. It upgrades the evaluation paradigm from "task completion" to "decision satisfaction" and converts subjective experience into objective, quantifiable metrics through a behavior‑chain evidence framework.

Core Methodology

The benchmark consists of three modules:

Dataset and Ground Truth : Implicit decision factors that affect user satisfaction are extracted from de‑identified interaction sequences to build a structured decision reference.

Dynamic Interaction Simulation : A UserAgent simulates users and replays real tool calls, allowing agents to be evaluated in authentic multi‑turn dialogues.

Full‑Chain Evaluation Metrics : From task execution to implicit demand fulfillment, multiple dimensions quantify an agent’s decision‑making ability.

Restore‑Identify‑Filter Pipeline

The benchmark proposes a three‑step "restore‑identify‑filter" method applied to large‑scale anonymized map service logs:

Restore : Reconstruct decision logic from the full interaction chain by linking preceding actions, current expressions, subsequent actions, and final task status. Continuous analysis and feedback attribution separate task‑progress signals from noise and locate unmet intent points.

Identify : Compare the restored complete demand with the explicit query to find implicit factors that are not expressed but would significantly narrow the solution space. These form a candidate set of hidden demands.

Filter : Retain only those implicit factors that can be traced to information available at decision time (current spatiotemporal context, prior actions, anonymized statistics). This ensures fairness by using only evidence visible to the model before answering.

Quantifying Satisfaction Impact

Each retained implicit factor receives a weight based on evidence support. The weight combines a long‑term preference probability (modeled by a three‑factor decomposition: preference intensity, recency, and momentum) with an instantaneous activation probability that reflects the current context. The final weight is the product of these two components.

Five‑Dimensional Truth Schema

Instead of a single correct answer, MapSatisfyBench defines a structured decision reference G(x) = (E, Z, T, C, R):

E(x) – Explicit Decision Constraints : Bounds derived from the literal query and spatiotemporal background.

Z(x) – Implicit Decision Constraints : Hidden factors with associated evaluation strategies, constraint types, and satisfaction weights.

T(x) – Tool Invocation Trajectory : Expected tool types, parameter constraints, and call order to ensure task executability.

C(x) – Clarification Rounds : Frequency of clarifications, measuring the system’s control of user cognitive load.

R(x) – Result Reliability : Consistency between generated content, tool output, and factual knowledge, preventing hallucinations.

These five dimensions drive the shift from "single‑point correctness" to "comprehensive satisfaction".

Truth‑Quality Control

A three‑stage closed‑loop quality control ensures reliability and reproducibility:

Automatic Generation : Large models annotate intent and structure from behavior‑chain signals.

Consensus Verification : Multiple independent LLMs evaluate consistency; low‑consensus samples are flagged.

Expert Review : Disputed samples undergo blind expert re‑annotation, and only those passing double verification are kept.

Deterministic Simulation Environment

UserAgent : When the evaluated agent asks a question, UserAgent provides the minimal sufficient answer derived from the ground truth, enabling natural multi‑turn evaluation.

Offline Sandbox : A cache of 22 real map‑service tool API responses guarantees reproducibility. If a cache miss occurs, an embedding‑based similarity search retrieves the closest cached record, preserving data authenticity while ensuring fair tool feedback across runs.

Key Findings

Using the React Agent framework, twelve mainstream LLMs (GPT, Claude, Gemini, DeepSeek, Qwen series) were evaluated on MapSatisfyBench.

Finding 1 : Models generally achieve high task‑completion rates (ECR > 0.85, GPT‑5.3 = 0.9272) but low implicit‑demand satisfaction (IISR ≤ 0.7170, Claude‑4.6‑Opus highest). Consequently, aggregated acceptance probability (AR) and satisfaction‑efficiency score (SES) are much lower, indicating difficulty in meeting hidden user preferences.

Finding 2 : All models have tool‑selection accuracy below 50 % and interaction efficiency below 0.5. Although anonymized preference summaries and historical interaction statistics are available, models rarely use them (e.g., POI search calls are 23 × more frequent than feature‑summary tool calls, 16 061 vs 691), preferring direct user clarification and increasing cognitive load.

Finding 3 : Enabling a "thinking mode" improves IISR for all model groups, with Gemini 3.1 Pro showing the largest gain, but even in this mode IISR remains far below ECR, demonstrating that longer reasoning chains alone cannot fully resolve satisfaction‑related decision challenges.

Conclusion

MapSatisfyBench pushes map‑agent evaluation from merely completing tasks toward assessing decision satisfaction within the user’s acceptance space. The main limitation of current LLM‑based map agents is no longer geographic knowledge or execution ability, but the inability to reliably recover implicit decision factors from user behavior chains. By providing a realistic, reproducible benchmark and a five‑dimensional truth framework, MapSatisfyBench aims to guide the next generation of satisfaction‑aware map agents.