From Street View to Navigation: Post-Training Multimodal LLMs for Reliable Landmark Extraction

Introduction

Amap (Gaode) launched a landmark-enhanced walking navigation mode that gives users intuitive cues like "turn right near the red-sign fast-food restaurant 50 meters ahead." To deploy this nationwide, the team built an end-to-end system that extracts landmarks from dash-cam imagery, judges their suitability for pedestrians, structures descriptions with orientation, distance, and visible frame ranges, and feeds them into the live navigation pipeline.

Traditional approaches relied on manual POI curation or object detection, which are costly, category-limited, and miss non-POI references such as fences or distinctive buildings. General multimodal models understand street scenes but poorly grasp navigation-specific selection criteria, often hallucinating colors, distances, or left/right relations. In navigation, a single wrong landmark can mislead a user, so precision is prioritized over recall.

1. From Object Recognition to Navigation Reference Extraction

Not every visible object qualifies as a landmark. A suitable landmark must be salient, easily distinguishable, describable in brief language, and visible along the relevant road segment. A distant shop sign may be recognized but hard to locate; a row of similar buildings may not help orientation. Continuous fences, prominent signage, or structurally unique buildings can be better references than named POIs.

The task requires simultaneous object discovery, suitability judgment, and attribute extraction. Input is a time-ordered dash-cam image sequence; output is a structured landmark set. Each landmark contains:

Description & Category : What the landmark is, its identifying features — provides a user-understandable, findable reference for voice guidance.

Orientation : Left, right, or ahead — supplies attribute basis for combining with road direction.

Distance : Near or far — helps users gauge the search range.

Visible Frame Range : Which frames in the sequence show the landmark — basis for later associating visibility with road segments.

When no suitable landmark exists, the model outputs an empty set. No bounding-box annotation is required, allowing targets with fuzzy boundaries (fences, hills) to be represented uniformly.

Business needs dictate optimization direction: one reliable landmark per segment can improve guidance, but a wrong color or left/right relation can mislead. Therefore the system prioritizes precision of output landmarks while retaining sufficient usable coverage, avoiding the trivial solution of always outputting an empty set.

2. Turning Navigation Expert Judgment into Training Data

Vague criteria like "salient," "easily recognizable," and "suitable for broadcast" must become executable annotation standards. Eight experienced navigation experts jointly developed guidelines through pilot labeling, disagreement review, and hard-case discussion.

The model first generates candidates; experts then add missing targets, correct descriptions, merge duplicates, and label each candidate as pass (usable) or reject (unsuitable). For rejections, experts record specific reasons, ultimately categorized into ten rejection types.

Annotation retains all qualifying landmarks in a scene rather than picking a single "best" one. This reduces annotator bias and lets the same data support multiple training signals: what to output, what to exclude, and which attributes to correct.

Temporal cropping is applied to image sequences: the latter half is kept, visible frame ranges are adjusted, and landmarks no longer visible are removed. This creates additional training samples under different observation windows and naturally yields empty-set samples where the original sequence had landmarks but the cropped version has none. The model learns to judge based on current input rather than memorizing scene-level answers.

3. Three-Stage Progressive Alignment: SFT → DPO → GRPO

On the same expert-annotated data, a progressive alignment pipeline is used: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO) with a navigation risk-aware reward (NARF). Each stage addresses basic capability, fine-grained selection, and overall risk trade-offs.

3.1 Multi-Task SFT: Learning to Extract per Business Requirements

The first stage establishes end-to-end extraction ability and jointly trains four auxiliary tasks: usability classification, salience ranking, visible range prediction, and description generation. For a candidate building, the model must not only generate "white building" but also judge whether it is sufficiently conspicuous, whether it could be confused with nearby objects, and in which frames it is visible. Synthetic reasoning examples in the main task follow a "discover candidates → analyze attributes and suitability → make selection" structure, helping the model learn the full judgment process. Natural empty scenes and cropped empty scenes provide supervision for outputting empty sets.

This stage solves "how to perform the task," but because expert annotations keep a relatively complete candidate set, the model may still tend to output too many targets or insufficiently reject plausible-but-unsuitable candidates.

3.2 Multi-Dimensional DPO: Learning to Distinguish Similar Answers

The second stage constructs preference pairs around concrete errors, covering seven comparison types: false acceptance, inaccurate orientation or visible range, false rejection, omission, and duplicate output. These pairs turn business judgments into comparable answer differences: for the same object, a description with correct orientation is preferred over one with swapped left/right; for the same sequence, an output that excludes an unsuitable target is preferred over one that includes it. DPO further reinforces these differences, correcting output preferences that remain after SFT.

3.3 Navigation Risk-Sensitive GRPO — NARF Reward: Aligning Output Policy with Navigation Cost

The third stage uses GRPO, letting the model generate multiple candidate answers during training and optimizing them via a reward that reflects navigation task costs. The key challenge: how to score an answer containing both correct and incorrect landmarks, and how it should compare to outputting an empty set.

Simple additive scoring (reward correct, penalize wrong) could encourage the model to output more targets to gain extra reward; a hard rule that any wrong answer must score below empty set would cause over-conservative behavior. The Navigation Risk-Aware Reward (NARF) categorizes outputs into three tiers:

Has output, no erroneous landmarks : Reward correct targets and their attribute quality → prioritize outputting reliable, accurately described landmarks.

Empty set : Slight penalty when usable landmarks exist; reward when none exist → actively abstain in appropriate scenes.

Output contains erroneous landmarks : Introduce penalty, discount, and cap total score → avoid masking errors by increasing candidate count.

NARF clearly separates error-free outputs from erroneous ones while preserving a limited exploration space between "erroneous output" and "empty set": only when enough correct predictions contribute can an erroneous answer's score exceed that of the empty set. This maintains precision priority while preventing the empty set from becoming the model's default easy choice.

Ablation experiments confirm the design necessity: a strict hard boundary achieved 96.80% precision but only 12.45% recall; NARF yields 92.29% precision and 45.52% recall. Reward design must observe both error rate and output capability, not precision alone.

4. Bounding-Box-Free Evaluation: Matching Landmarks and Verifying Attributes

Training iteration depends on evaluation that reflects real business errors. Without bounding boxes, standard detection IoU cannot be used; pure text similarity is insufficient because "left red building" and "right red building" are textually similar but convey opposite navigation instructions.

A hybrid bipartite matching approach compares each predicted landmark with expert annotations on attributes. Matching proceeds in two steps: first, explicitly incompatible pairs are filtered out (category mismatch, left/right conflict, non-overlapping visible frame ranges, explicit single-color conflict, low semantic similarity). For remaining pairs, a composite score combining semantics, category, distance, range, orientation, and color is computed, and the Hungarian algorithm solves a one-to-one matching. Each annotation target matches at most one prediction, so duplicate descriptions are not repeatedly counted as correct.

Visible range scoring reflects business needs: a conservative prediction within the true visible range carries different risk than extending the range to where the target is actually invisible. The evaluation distinguishes these cases.

Matching results further differentiate two error types: FP-Rej (matched to an expert-rejected target) indicates suitability judgment bias; FP-Hal (unmatched to any annotation) is treated as hallucination under the current protocol, though it may involve attribute conflicts or annotation omissions, requiring manual review of images and match details.

The same matching method serves both offline evaluation and GRPO reward calculation, ensuring consistent standards between training optimization and effect assessment. The evaluation system retains attribute scores and match relations, helping pinpoint whether issues lie in target selection, description, orientation, or visible range.

5. Business Integration: Offline Landmark Production, Online Retrieval and Broadcast

The trained model enters the navigation business via offline batch production. The full pipeline: road segment and image filtering → image sequence extraction → model generates structured landmarks → result verification and release → retrieval by road segment → compose navigation broadcast.

5.1 Offline Side: Turning Model Inference into Reusable Road Data

Production first selects road segments meeting length, curvature, and data quality requirements, then runs batch inference. The deployment uses an 8B/32B model cascade: run the 8B model first; only when it outputs an empty set is the 32B model invoked to supplement coverage on harder scenes while controlling large-model invocation scope.

In a batch of 500,000 qualified road segments with imagery, 411,000 produced at least one valid landmark (82.2% coverage). Results are organized by road segment ID and written into a KV index for repeated use by the navigation service. Coverage is measured against filtered imagery-available segments.

Ongoing production manages result lifecycles. The engineering pipeline maintains per-segment states: pending, imagery-but-no-result, valid-result, and manually taken down. This supports re-computation, result updates, and problematic landmark removal. Model empty-set outputs become an explicit data state, distinguishable from missing imagery.

5.2 Online Side: Integrating Landmark Attributes into Existing Guidance Flow

Online navigation retrieves landmarks by road segment ID and fills attributes (description, orientation, distance) into broadcast templates. Example: "Do you see the black fence? After departure, it should be on your right not far away."

This path places LLM inference offline, so online requests do not wait for visual model generation. If no usable landmark exists, the system falls back to pure direction instructions. Landmark data supports departure direction confirmation, straight-ahead confirmation, turn reference, and post-turn direction confirmation; the reported online A/B results focus on the departure scenario.

This integration converts model capability into retrievable road attributes, allowing model training, data updates, and online broadcast to iterate independently.

5.3 Quality Management: Evaluation Metrics Combined with Release Spot-Checks

Offline metrics compare models; pre-release checks examine actual production results. Before each city batch release, experts spot-check 100–500 cases, requiring a case-level pass rate above 95%; a single erroneous landmark in a case fails it.

Online monitoring retains error broadcast reports, complaints, and safety event tracking, and supports taking reported inaccurate landmarks offline from the index. Thus model evaluation, batch spot-checks, and online correction jointly uphold data quality. Since experiments began, among over 10,000 user feedback items, fewer than 10 concerned landmark accuracy, and no safety events attributable to landmark broadcasts were observed.

6. Effect Verification: Model Quality, Road Coverage, and User Behavior

Effectiveness is validated at three levels: model output reliability, offline road segment coverage, and user departure speed.

6.1 Offline: Progressively Improving Landmark Precision

On 240 held-out image sequences, Qwen3-VL-32B results across training stages (values in %):

Model Stage          Precision  Recall  FP-Rej  FP-HalNo task training     49.55      31.09   7.59    42.86SFT                 83.45      64.29   4.73    11.82DPO                 87.58      58.26   3.37    9.05GRPO (NARF)         92.29      45.52   2.48    5.23

SFT established task extraction capability; DPO and GRPO further improved output reliability. From final SFT to NARF, precision rose 8.84 percentage points, FP-Hal dropped from 11.82% to 5.23%, while recall decreased — reflecting the precision-first training trade-off.

As a baseline, several closed-source multimodal APIs achieved only 56%–62% precision zero-shot, with hallucination rates over 30%; even open-source models with 7× more parameters showed no fundamental improvement. General street-view understanding alone cannot directly support broadcast; training navigation standards into the model is essential.

Recall measures how many expert-annotated targets the model recovers; road-segment coverage measures whether a segment has at least one valid landmark. They answer different questions and come from different evaluation sets. Business decisions require combined observation to judge whether the model reduces false positives while retaining enough usable data.

6.2 Online: Faster Stable Walking After Departure

From April 20 to May 20, 2026, a 30-day user-level A/B experiment ran in Beijing, Shanghai, Hangzhou, and Shenzhen. The treatment group had ~2.02 million users who initiated at least one navigation. During the experiment, ~40% of treatment-group departure events played landmark-enhanced instructions; the rest used the same pure-direction instructions as the control group.

The primary metric was "median time from end of departure broadcast to stable walking onset," measuring how quickly users confirm departure direction.

City          Control Group  Landmark Strategy Group  Relative ReductionBeijing       7.46 s         5.73 s                   23.2%Shanghai      7.77 s         6.17 s                   20.6%Hangzhou      7.47 s         5.77 s                   22.8%Shenzhen      7.44 s         5.37 s                   27.8%Four-City Total 7.54 s         5.77 s                   23.5%

The four-city median time decreased by 1.77 seconds, a 23.5% relative reduction (95% CI: 22.7%–24.3%). This result reflects the overall strategy including landmark broadcasts and pure-direction fallbacks; all four cities showed improvement.

For users, landmark prompts provide a reference that matches the visible environment; for the system, this metric connects offline model quality to real navigation behavior: the ultimate test of model improvements is whether users take their first step faster.

7. Future Iterations

Future work will continue iterating on landmark data usability and production efficiency: refine expert judgment standards to reduce annotation cost for new cities and scenarios; iterate data and training strategies based on error types; optimize small-model extraction capability and cascade mechanisms to control large-scale production costs.

The overall solution threads landmark selection standards through data annotation, multi-stage training, attribute-level evaluation, and online service, realizing a complete pipeline from street-view reference extraction, road data production, to navigation prompt generation.

References

[1] Xin Yang, Ziqi Cao, Changning Shao, Junhao Lin, Fanxiang Zeng, Yue Liu. Silence over Error: Risk-Averse Navigation Landmark Extraction via Progressive MLLM Alignment. EMNLP 2026 Findings.

[2] Rafael Rafailov, et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.

[3] Zhihong Shao, et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024.

[4] Nicolas Carion, et al. End-to-End Object Detection with Transformers. ECCV 2020.