DIANOIA: Multi-Agent Diagnosis & Repair for Navigation Tool Trajectories

Problem: Tool Dependency in Navigation Training Data

Amap Navigation Live mode fulfills user requests (e.g., "find a coffee shop along the route") through a sequence of tool calls: route reading, along-route search, candidate filtering, result return. Each step depends on current location, route, and previous tool outputs. Training data must provide complete, executable tool-call sequences that satisfy user intent, tool interfaces, and execution environment. Real interaction logs lack ground-truth call sequences; synthetic data fills this gap but introduces errors such as missing prerequisite queries, incorrect variable references, premature "task completed" markers, and empty-result handling failures. These errors often affect only part of a trajectory, making holistic scoring insufficient for targeted repair.

Why Multi-Agent for Data Production

Separating Exploration from Verification

Data quality correlates with the model capability used during production. Multi-agent systems effectively coordinate and probe LLM upper bounds. Generation (exploring feasible paths), verification (tracing tool dependencies), and repair (synthesizing local fixes) have distinct concerns; explicit role separation allows tailored constraints and evidence per role. For example, a Minimalist proposer seeks shorter call chains, a Skeptic checks preconditions, an Explorer tries alternatives. The same failed trajectory thus receives diverse repair suggestions. Offline batch processing permits extra compute on hard samples; repaired trajectories feed downstream training without requiring multi-agent inference at serving time. Navigation tasks also offer executable tool environments, giving reviewers concrete execution feedback (actual returns, failure points, parameter validation) for grounded judgments.

Gaps in Existing Multi-Agent Approaches

The authors surveyed representative methods:

Self-Consistency / Best-of-N : multiple candidates increase chance of a valid solution, but text voting fails because valid sequences have multiple syntactic forms; cannot repair steps missing in all candidates.

ReConcile (discussion/consensus) : models exchange rationales, but real execution evidence must be injected; agreement still cannot replace tool-dependency and task-completion checks.

Mixture-of-Agents (hierarchical synthesis) : aggregates prior outputs, yet synthesized trajectories need variable rebinding, dependency re-validation, and re-execution; text quality alone is insufficient for acceptance.

AgentCoder (execution-feedback collaboration) : separates generation, test design, test execution; however, programming test harnesses must be replaced by navigation tool environments with location, route, and user-intent acceptance criteria.

The key contribution is organizing these capabilities into a pipeline suited for navigation data production: diverse candidates, execution reports enabling local diagnosis, review opinions directly usable for repair, and independent verification gating re-entry. Three diagnostic channels—Coverage (candidate diversity), Fidelity (evidence reliability), Synthesis (evidence utilization)—guide iterative improvement.

Technical Framework: Diagnosis, Execution, Review, Repair

Three Channels for Bottleneck Localization

Coverage / Exploration : whether candidate set contains a feasible solution. If multiple proposers fail at the same step, adding similar candidates has diminishing returns; role prompting reduces error correlation.

Fidelity / Information : whether the system can pinpoint correct vs. incorrect steps. Tool execution, parameter validation, and dependency checks provide strong signals; user-intent satisfaction requires contextual evaluation. Downstream modules must know signal provenance.

Synthesis / Aggregation : how candidates and evidence are combined. When a viable candidate exists, synthesis must preserve it; when all fail, it attempts to stitch locally correct steps with diagnostic feedback.

These channels dictate debug order: no feasible path → improve exploration; feasible path exists but undetected → check verification signals; diagnosis clear but final output fails → inspect synthesis logic.

Four-Stage Information Flow

Stages pass candidate trajectories, execution reports, and structured diagnoses. The synthesizer decides which steps to retain, which preconditions to add, and which results need re-validation.

Propose (Candidate Generation) : Input: user request, navigation context, tool constraints. Output: multiple candidate call sequences. Three proposers: Minimalist (concise chains), Skeptic (preconditions & stepwise verification), Explorer (alternatives). Hard constraints (valid location sources, variable reference rules, duplicate-call splitting, empty-result handling) apply to all roles.

Execute (Real Execution) : Input: candidate sequences, tool environment. Output: tool returns, successful steps, failure locations, error messages. Execution success only means the call ran; it does not guarantee the result satisfies the user request. Execution status and task-completion judgment are kept separate.

Review (Cross-Examination) : Input: candidates + execution reports. Output: retainable fragments, issues, risks, modification suggestions, scores. Reviewers check POI parameter sources, variable references, required fields, tool dependencies, termination semantics.

Synthesize (Repair & Synthesis) : Input: all candidates, execution reports, review opinions. Output: repaired sequence, re-entered into execution and final verification. A failure-pattern library supplies additional constraints (fabricated POI names, parameter type errors, missing along-route search steps). For local errors, the valid prefix is kept and subsequent steps modified. The new sequence must re-execute because prior returns, variable bindings, and later inputs may have changed.

Worked Example: Missing POI Coordinate

Initial candidate: directly calls route planning without first querying destination POI coordinates. Execution: fails at route planning, logs missing coordinate parameter. Review: retain route-planning intent; add POI search step; verify search returns valid coordinates. Synthesis: new sequence queries POI, binds valid coordinates to route-planning parameters, adjusts downstream references. Final verification: re-executes full sequence, checks final result and task-completion marker against actual state. If search yields no usable result, the system falls back to alternative queries or clarification paths per tool constraints.

Pipeline Integration: Compute on Demand

Confidence Pre-filtering & Re-entry

Existing pipeline uses multiple low-cost models to evaluate synthetic samples. High-confidence samples proceed directly to the pool; low-confidence samples enter DIANOIA for refinement. Pre-filtering allocates compute: low confidence signals need for deeper inspection, not that the whole trajectory is wrong. After refinement, the new target sequence undergoes independent verification (parameter parsing, tool encoding legality, termination semantics). Only verified samples re-enter the pool; failures retain original sample and diagnostic logs for analysis.

Concurrency, Recovery, Write-back

Candidate generation and cross-review (once execution reports ready) run in parallel, with unified rate limiting to control model-service load.

Batch jobs persist progress for crash recovery; single-stage failures roll back without affecting the whole batch.

Final write-back is gated by independent verification; synthesizer scores only assist, not replace, this gate.

Metrics for Pipeline ROI

Low-confidence salvage rate : (refined samples passing final verification) / (samples entering refinement). Measures whether repair capability translates to usable output.

Cost per new qualified sample : (total refinement pipeline cost) / (new qualified samples from refinement). Denominator uses new qualified samples, not processed samples, to distinguish effective repairs from wasted retries.

Spot-check pass rate & error-type distribution : fixed-protocol sampling of re-entered samples to uncover auto-verification blind spots and error migration.

Technical Validation & Pipeline Value

Reproducible Task Benchmarks

Evaluated on GSM8K (math), MBPP (code generation), BFCL-SP (function calling) using Qwen3-30B-A3B-Instruct-2507 as base model. DIANOIA outperformed representative baselines:

GSM8K: ReConcile 89.8% → DIANOIA 91.1% (+1.3 pp)

MBPP: Best-of-3 (same verifier) 78.4% → DIANOIA 84.6% (+6.2 pp)

BFCL-SP: Two Heads 88.8% → DIANOIA 92.3% (+3.5 pp)

Token efficiency: on MBPP, MoA reached 84.2% at ~7.73M tokens; DIANOIA reached 84.6% at ~1.54M tokens (~1/5 tokens). On MBPP, 5.5% of DIANOIA's correct outputs came from cases where all initial candidates failed execution, demonstrating that execution evidence and review opinions can generate novel viable solutions.

Comparison with Stronger Single Models

DIANOIA (Qwen3-30B) vs. Qwen3-235B-A22B and DeepSeek-V3.2 (single-model, zero-shot, no extended reasoning):

GSM8K: 91.1% vs. 86.4% / 85.3%

MBPP: 84.6% vs. 80.2% / 81.2%

BFCL-SP: 92.3% vs. 89.0% / 83.5%

On BFCL-SP (closest to navigation tool calling), DIANOIA exceeds Qwen3-235B by 3.3 pp and DeepSeek-V3.2 by 8.8 pp. This shows that a smaller base model plus diagnostic/repair pipeline can surpass larger single models, offering an alternative to model upgrades. Note: DIANOIA incurs multiple model calls; cost/latency trade-offs require separate measurement.

Stronger Base Models Still Benefit

Applying the same pipeline to stronger bases (Qwen3.6-35B, DeepSeek-V4-Flash, Gemini-3.5-Flash) on full MBPP (500 problems) yielded consistent gains over single-model and Best-of-3:

Qwen3.6-35B: single 78.4% → Bo3 83.2% → DIANOIA 88.8% (+10.4 pp vs single)

DeepSeek-V4-Flash: single 88.8% → Bo3 97.0% → DIANOIA 98.2% (+9.4 pp)

Gemini-3.5-Flash: single 93.0% → Bo3 93.8% → DIANOIA 96.8% (+3.8 pp)

Base capability and process improvement stack. However, marginal gain over simple sampling shrinks as base strength increases (DeepSeek-V4-Flash Bo3 already 97.0%, DIANOIA +1.2 pp with overlapping confidence intervals). Production deployment must weigh this marginal gain against extra compute.

Capabilities Gained in Data Production

Low-confidence samples gain an execution-verified re-entry channel. Partially correct trajectories retain valid steps; repaired ones become new training data; failures leave concrete evidence.

Diagnostic logs (candidates, execution reports, reviews) enable root-cause analysis of parameter errors, missing dependencies, termination issues, guiding prompt and verification-rule improvements.

Pre-filtering concentrates multi-agent compute on needy samples, reducing overhead on high-confidence ones. Unit qualified-sample cost depends on salvage rate, tool-call fees, and verification cost—jointly evaluated.

Downstream impact on navigation agent performance requires controlled training comparisons and business metrics.

Future Directions

Finer-grained resource allocation : beyond confidence-based routing, distinguish candidate insufficiency, execution anomalies, semantic uncertainty; allocate extra candidates, stronger verification, or adjusted repair loops per case. Tune candidate count, review count, repair rounds by marginal returns.

Extend to more travel toolchains : walking, bus, subway transfer involve different preconditions and termination criteria; requires re-engineering tool contracts, failure modes, acceptance rules.

Lightweight candidate generation & review : control batch production cost.

Shift metrics from "how many generated" to "which samples verifiable, which errors repairable, cost per new qualified sample" to guide investment in model, tooling, or data-process improvements.

References

[1] Wang et al., Self-Consistency Improves Chain of Thought Reasoning, ICLR 2023. [2] Chen et al., ReConcile: Round-Table Conference Improves Reasoning via Consensus, ACL 2024. [3] Du et al., Improving Factuality and Reasoning via Multi-Agent Debate, ICML 2024. [4] Huang et al., AgentCoder: Multi-Agent Code Generation with Iterative Testing, arXiv:2312.13010. [5] Cobbe et al., Training Verifiers for Math Word Problems, arXiv:2110.14168. [6] Austin et al., Program Synthesis with LLMs, arXiv:2108.07732. [7] Patil et al., Berkeley Function Calling Leaderboard (BFCL), ICML 2025.