How Amap’s Evaluation Team Measures Product Success: Roles, Methods, and Tools

Introduction – In the fast‑growing internet era, companies create specialized roles to improve product effectiveness and user experience. At Amap (Gaode), the Evaluation team is dedicated to assessing product impact before launch and after release, building a three‑dimensional evaluation system.

Who Is the Evaluation Team?

The Evaluation team functions as the product‑effect assessment group, aiming to verify requirements from a user perspective, analyze internal data, user data, and competitor data, and construct a comprehensive evaluation framework.

Why Does Evaluation Exist?

Product updates raise questions about performance, strategy impact, and whether implementation matches product‑manager and user expectations. The team provides quantitative methods to detect differences, confirm positive effect, and protect user experience.

How Evaluation Is Conducted

Typical techniques include offline evaluation, AB experiments, online metric monitoring, issue analysis, competitor monitoring, and road‑testing.

1. Offline Evaluation

Before launch, the team validates product requirements, determines whether standards are met, and identifies major issues. Core activities involve establishing collaboration processes, building evaluation expertise, and developing tools.

Collaboration Process – Mirrors a version‑development workflow: requirement clarification, development, testing, and release. Evaluation joins at the requirement‑analysis stage, defines evaluation plans, checks tool readiness, collects data, validates results, and issues a report.

Evaluation Plan – Based on the impact scope, the plan defines sample selection, evaluation methods, and standards.

Evaluation Samples – Samples are split into random corpora and targeted corpora. Targeted samples focus on specific dimensions, while random samples reflect overall impact. When possible, both are used together.

Evaluation Standards – Distinguish between absolute truth (e.g., ground‑truth data) and relative truth (e.g., user logs, click behavior). The presence, accessibility, and automation potential of truth data guide standard selection.

Evaluation Methods – For tasks with absolute truth, automated evaluation is feasible; otherwise, manual or semi‑automatic assessment is required.

Manual Evaluation

Human judges score product quality, a practice used by Google, Microsoft, Baidu, Apple, etc. Benefits include early issue detection, alignment with quantitative metrics, comprehensive quality definition, and detailed user‑feedback insight. Drawbacks are high cost, limited coverage, and lower efficiency.

Key success factors are clear standards, robust processes, and supporting tools. Documentation should be simple, exhaustive, regularly updated, and maintained by dedicated owners.

Quality control mechanisms such as multi‑rater consensus, blind review, and tiered review (initial vs. senior reviewer) are employed.

Evaluators can be ordinary users (crowdsourced) or experts; expert evaluation is common for specialized domains like navigation.

Evaluation Tools

Tools ensure efficiency and quality, offering data warehousing, task management, data collection, diff analysis, result visualization, sampling, assignment, and automated reporting. Custom task types, scoring schemes, and case formats can be defined to fit specific business needs.

Automated Scoring Models

Machine‑learning models learn from manual evaluation features to generate GSB scores, providing an auxiliary judgment for evaluation tasks.

Smoke Testing

Core scenarios and metrics are defined, acceptable variance thresholds are set, and severe cases are identified. This enables rapid validation for experiments and can automate release decisions.

Metric Analysis + Anomaly Detection

For domains without absolute truth, a best‑practice approach combines overall metrics, scenario‑specific metrics, and anomaly indicators. After detecting abnormal cases, manual verification determines final conclusions.

Road Testing – The ultimate validation for navigation products, involving real‑world user experience, remains costly but indispensable.

2. AB Experiments

When model tuning or other changes require online observation, the team moves from offline validation to AB testing. The core pipeline includes traffic splitting, metric observation, and scientific result generation. Building a robust AB framework is a long‑term effort.

3. Online Validation

After successful offline and AB validation, the feature is fully released. Post‑launch monitoring analyzes key metrics, user feedback, and anomaly detection to confirm expected benefits and spot any unexpected changes.

Conclusion

The construction of an evaluation system mirrors the building of a comprehensive product‑effect assessment framework. While the specific title may vary across companies (testing, product, operations), the core responsibilities remain essential. At Amap, the dedicated Evaluation role reflects a strong focus on user experience and product impact, and the system continues to evolve toward making travel better.