Meituan's Agent Evaluation System: From Scoring to Infrastructure Capability

What Is Agent Evaluation

The core purpose of evaluation is to answer whether an agent is good or bad, and exactly where it succeeds or fails, so as to guide the next iteration. Evaluation serves as a "precision instrument" for agent effectiveness and must support real‑business R&D, release, regression, optimization, and scaled deployment — not just offline leaderboards or one‑off demos. The foundation is observation, yielding the formula: Observation + Evaluation = Continuous Iteration .

Evaluation methods evolve with AI form factors through three stages: traditional ML answers "how accurate"; LLM evaluation answers "how capable"; agent evaluation answers "when the model is embedded in a real system with prompts, skills, toolchains, memory, state management, and business processes, can it stably deliver good results?" The evaluation object shifts from a single model to a complex "model + system + tools + process" system.

Two agents may both produce the correct final answer, yet differ drastically in engineering value: one follows a clear, stable, reproducible path with controlled latency and tool calls; the other relies on trial‑and‑error, chaotic paths, and lucky hits. Judging only the final output misclassifies them as equal. Therefore agent evaluation must cover four layers:

Result layer : task completion, output usability

Process layer : planning rationality, step stability

Efficiency layer : latency, token consumption, tool‑call count

Risk layer (safety) : privilege escalation, mis‑operation, security hazards

Since GPT's rise in 2023, agent development has progressed from chatbots to long‑horizon frameworks like Claude Code and OpenClaw, and evaluation is moving from "answer evaluation" to "behavior evaluation".

Why Observation Is the Cornerstone

Agents belong to the broad SaaS layer; LLMs provide generalization but introduce randomness, while users expect stable, reliable agents. To bridge randomness and reliability we must adopt an engineering view: problems that cannot be seen are almost impossible to solve stably . An agent execution typically involves a chain of user input → prompt construction → model reasoning → tool calls → intermediate results → final output. A failure at any layer degrades the final result. Stable results require stable processes, but if logs only capture "what the user said" and "what the agent replied", root‑cause analysis is impossible. This "I want to see the case but find no logs" pain point drove the industry to build trace systems that fully expose internal reasoning and record all inputs influencing model output. Precise observation of every "invisible action" is the necessary path from probabilistic generation to industrial‑grade reliability.

Trajectory Evaluation vs. Response Evaluation

Agent evaluation essentially answers "how good" and points the iteration direction. It must attend to both the response (final output) and the trajectory (trace or full execution path).

Core Methodology of Evaluation

1. The Core of an Evaluation System Is Not Stacking Metrics but Building Bridges

Agent evaluation systems must pursue interpretability between business value and evaluation metrics. A natural gap exists between model capability metrics and business outcome metrics; they cannot map directly. A layer of task‑system‑oriented bridge metrics is required. A layered approach:

Business layer: DAU, retention, click‑through

System layer: recall rate, click‑through rate

Agent layer: intent recognition accuracy, retrieval effectiveness, result integration credibility

Only by linking these layers can we answer "why business metrics dropped" and "why model improvements didn't translate to business gains". This requires people who truly understand the business process to co‑create the metric system.

2. Objective and Subjective Evaluation in Parallel

Industry practice borrows heavily from LLM evaluation, splitting into objective and subjective evaluation. The realistic approach:

Use objective evaluation for high‑frequency, structured, rule‑based parts

Use subjective evaluation for open‑ended, high‑value, complex business scenarios

Calibrate objective and AI evaluation with subjective evaluation, then hand the scalable portion to automation

3. Aligning Subjective Evaluation via "Human‑Human Consistency" and "Human‑Machine Consistency"

"Good or bad" is subjective; standards must be aligned, otherwise we cannot distinguish metric noise from real improvement. The real difficulty is not "no one can evaluate" but "different people evaluate differently, and machines differ from humans". Over the past year, Meituan's Turing team found multiple business units falling into the same traps. Their key insight: human‑human alignment and human‑machine alignment .

Human‑human consistency : One "dictator" is better than ten "democrats". A strong role aligns product, operations, R&D, and QA to a single evaluation system, avoiding fragmentation. Different annotators align via back‑to‑back labeling.

Human‑machine consistency : Machine evaluation results must match human results; otherwise they are not trustworthy. The goal is scaled efficiency for larger business volumes.

Alignment practice: drill fuzzy metrics into finer rubrics (dimensions), then make each rubric as binary as possible (yes/no/unknown, 0/1/unknown). Steps:

Metric drill‑down : decompose vague concepts into clear dimensions

Rubric binarization : converge scoring rules to binary judgments

Continuous iteration : use unknown‑rate to audit rubric quality until per‑rubric human‑human and human‑machine agreement reach trustworthy thresholds (e.g., 85%, 90%)

This turns "subjective fuzzy feelings" into "judgeable factual evidence". Applying this method, the Digital Stationmaster business achieved 99% human‑machine consistency; Beam adopted Turing's binarization and lifted consistency from 62% to 92%.

Case Studies

Case 1: Evaluating a middle‑school essay (max 40 points) – The article shows a rubric table breaking the essay into multiple binary dimensions (structure, argument, language, etc.) instead of a single holistic score.

Case 2: Rider outbound call – "Is the model reply colloquial?" – Classic error: "Judge if the reply is colloquial, score 0‑10." Improved binarized rubrics:

Does the model address the rider as "您" (formal you)?

Does it use non‑official colloquial words like "甭客气", "明儿见"?

Does the output contain modal particles like "吧", "呢", "那个"?

Annotation is an action; evaluation is a goal‑oriented judgment process. Machine pre‑annotation aids human efficiency, but only when human‑machine consistency is guaranteed does it become automated evaluation; otherwise it is merely machine annotation.

4. Agent Evaluation Is a Practical Science

From an execution perspective, agent evaluation decomposes into five key stages:

Collection : gather raw task data from production or sandbox

Cleaning : deduplicate, categorize, supplement context, fix dirty data

Evaluation : human evaluation, AI evaluation

Quality Check : verify evaluation standard stability, result credibility

Analysis/Attribution : locate root causes, produce optimization suggestions and regression tasks

These five stages, together with online A/B testing and continuous observation, form the agent iteration data flywheel.

Most new teams mistakenly design a complex, sophisticated metric system upfront; complex metrics are harder to execute and align. Agent evaluation is a practical science. At the start, "getting the data flywheel running efficiently" matters far more than "designing a perfect evaluation system". The system grows by feeding on good and bad cases. Best‑practice path:

Start from high‑frequency core scenarios, define a few key metrics

Collect bad cases from production

Accumulate high‑quality good cases to define "what good looks like"

Convert good/bad cases into standard evaluation samples

Feed evaluation results back into prompt, skill, strategy, and model optimization

Continuously resample from new online performance for the next iteration

Bad cases often have higher value because they expose capability boundaries and system weaknesses; good cases help define the paradigm of high‑quality completion. A mature evaluation team's core ability is not building a perfect system from day one, but continuously turning online issues, failure samples, and vague feedback into structured evaluation assets. For the Digital Stationmaster business, the project began with 20‑odd metrics and expanded to nearly 200 after a year.

5. Expert Knowledge Supplements Model Gaps in Vertical Domains

Despite base model leaps since GPT‑3.5, model capabilities are not omnipotent. Many "wishful" demands ignore that models are trained on token probability distributions; even with emergence at scale, capability gains still depend heavily on high‑quality corpus input (RLHF, DPO, GRPO all rely on it). ByteDance's Xpert crowdsourced expert labeling platform produces high‑quality data for geography, code, law, medicine to support Doubao base model training. When facing scarce business‑domain knowledge (or no public high‑quality data), injecting industry expert knowledge becomes the key breakthrough — especially during cold start.

Who defines "good"? The industry experts who know the business best and have the strongest sense.

6. Summary & FAQ

This chapter distills the core methodology from the Fulfillment project and a year of BP (business partner) engagements across Meituan.

Q1: Must the "dictator" be a single person? A team must follow one evaluation system. The dictator consolidates opinions and makes final calls when stakeholders cannot align, preventing fragmentation and internal friction.

Q2: Does relying on a "dictator" carry risk? Evaluation systems evolve continuously. As business scales from early adopters to all users, user profiles shift, requiring evaluation targets to adjust. The dictator's value is standard alignment; targets are corrected by real‑world bad/good cases. The initial system should be crafted by those who know the business best.

Q3: Is a seed evaluation set mandatory for cold start? Can't we just launch a small‑traffic canary and collect cases? It's a risk‑cost trade‑off. Seed sets are recommended because LLM randomness can cause severe experience shifts on corner cases; regression testing protects baseline capability and incrementally widens the capability envelope with good/bad cases. If the scenario tolerates errors well, or seed‑set construction cost far exceeds negative feedback from live trials, a small‑traffic launch is acceptable. Recommended: humans produce a small seed set, then AI assists in generation/expansion for low‑cost cold start.

Q4: What if invited experts disagree on "good"? Invite a batch of experts, extract the commonalities to build the evaluation system (e.g., top salespeople defining excellent sales SOP). Non‑common parts are not waste — they indicate multiple valid strategies. Convert disagreements into distinct agent styles or strategy branches (e.g., aggressive vs. steady AI telemarketing) and evaluate them independently on separate benchmarks. These divergence points become nutrients for fine‑grained iteration and long‑tail coverage.

Evolution of Agent Observation and Evaluation

2023: GPT explosion; 2024: workflows; 2025: Claude Code; 2026: "Lobster" (OpenClaw) and "Hermès" heat — long‑horizon agents enter mainstream. Agent harnesses fully enter the long‑horizon era.

Long‑horizon agents differ from short‑horizon ones in how they handle "complexity brought by time span": multi‑step task decomposition, repeated tool/skill calls, reading intermediate results and dynamically adjusting strategy.

1. Short‑Horizon Agent Observation & Evaluation

Typical I/O: Query → Answer. Agents mostly "answer questions" with few system operations, not "enter OS to execute tasks". Typical scenarios: AI search, customer service bots. Evaluation focuses on the answer itself (accuracy, relevance, tone, safety, format). Over the past year Turing built mature solutions covering human and machine evaluation.

2. Long‑Horizon Agent Observation & Evaluation

1) Paradigm Shift Brought by Long‑Horizon Agents

Long‑horizon agents solve "complete a complex task", not "answer a question". They require multi‑step decomposition, multiple tool/skill calls, and dynamic strategy adjustment based on intermediate results.

Recall the goals:

Observation goal : reconstruct the scene, pinpoint issues, solve "I want to see the case but find no logs"

Evaluation goal : answer "how good is the agent", point iteration direction

2) Skill Evaluation

Based on research into the post‑Spring‑Festival 2026 Lobster/Skill surge, demand comes mainly from broad "operations efficiency" scenarios, falling into three user groups (merchant side exploring/piloting; internal AI‑at‑Work covering product, ops, R&D, sales; skill development barrier dropping, even AI‑assisted generation). Conclusion: future evaluators are not just a few product/ops/R&D folks, but potentially everyone who creates, modifies, or integrates skills. This demands evaluation systems that are simple, standardized, automated, and integrable into dev/release pipelines.

Core pain point: people don't know how to write good skills, and lack tools for full‑lifecycle skill evaluation. Skill lifecycle decomposed into: definition → development → testing → release → monitoring → iteration.

Skill evaluation pain points split into three aspects (illustrated in article).

Task‑Oriented Evaluation

Jan 9, 2026: Anthropic blog "Evaluating AI Agents" introduced task‑oriented long‑horizon evaluation. A Task is defined as "a single test with clear input and success criteria". Combining Anthropic and open‑source definitions:

prompt : defines the problem/request

expected_behavior : defines the expected agent behavior

Send prompt in prod/test env, obtain real execution trace via trace system

Form tuple (prompt, expected_behavior, trace) — analogous to short‑horizon (query, ground_truth, answer) — enabling evaluation

3) Differences Between Long‑Horizon and Short‑Horizon Evaluation

Summary table in article contrasts: evaluation object (answer vs. task system), core metrics (text quality vs. completion rate + trajectory quality), evaluation method (static reference answer vs. dynamic behavior verification), observation granularity (input‑output only vs. full trace), automation difficulty (lower vs. higher), and infrastructure needs (simple diff tools vs. sandbox, replay, regression gates).

Essential change: ChatAgent evaluation cares "how well it speaks"; long‑horizon agent evaluation cares "whether the task got done, and how it got done".

4) Shift from Human‑Led to Machine‑Led Evaluation

ChatAgent era: core evaluator alignment → outsourced alignment → machine alignment.

Long‑horizon scenario: can shorten to core evaluator alignment → machine alignment → scale.

Reasons:

Richer execution trajectories, high information density makes humans the bottleneck

Low skill production barrier, explosive quantity growth makes large‑scale human labeling unsustainable

Continuous base model breakthroughs

This doesn't diminish human importance; rather:

Humans should design high‑value standards and align rubrics

AI should handle scaled execution, initial screening, regression verification

Platforms should own precipitation, replay, alerting, attribution

AI evaluation amplifies the core evaluator's judgment standards, not just "machine scoring".

5) Minimum Capabilities for Long‑Horizon Agent Evaluation Infrastructure

To support large‑scale agent and skill ecosystems, evaluation infrastructure must provide:

Full‑chain replay : reproduce a task from input to result

Case management : unified maintenance of task samples, context, constraints, rubrics

Execution sandbox : layered isolation (read‑only, writable, high‑risk)

AI evaluation engine : rubric‑driven human‑machine alignment and auto‑scoring, simple enough for all skill creators

Reporting & attribution : not just scores, but pinpoint whether issues lie in planning, tools, environment, or skills

Regression mechanism : auto‑trigger historical case regression on version upgrades

Gatekeeping : embed evaluation results into development, release, and operations workflows

Without these, evaluation stays at "ad‑hoc analysis" and "project‑level support", never becoming a production system component.

Summary: Agent Evaluation Moving from Scoring Action to Infrastructure Capability

Two key shifts:

Evaluation object changed : from "answers" to "task systems". We now care about capability, stability, efficiency, and risk across the full execution chain, not just output content.

Evaluation method changed : from Query→Answer text quality evaluation to Prompt→Expected Behavior behavior evaluation. Ground truth is no longer always unique; process quality, task completion, and trajectory quality become core objects.

What truly matters is not running a few evaluations, but building a system that makes problems visible, standards clear, scale runnable, processes connectable, and iterations drivable .

Appendix: Industry Research on Open‑Source Long‑Horizon Agent Evaluation

Key definitions from the open‑source community:

Task (Problem/Test Case) : single test with clear input and success criteria; each attempt is a trial; multiple trials run for consistency.

Grader : logic evaluating an aspect of agent performance; a task can have multiple graders, each with multiple assertions (checks).

Transcript (Trace/Trajectory) : complete trial record including outputs, tool calls, reasoning, intermediate results, interactions.

Outcome : final environment state at trial end (e.g., SQL record existence for a flight booking agent).

Evaluation Harness : end‑to‑end infrastructure running evaluations — provides instructions/tools, concurrent task execution, step recording, scoring, result aggregation.

Agent Harness (Scaffold) : system enabling a model to run as an agent — handles input, orchestrates tool calls, returns results. Evaluating "an agent" means evaluating harness+model together (e.g., Claude Code is a flexible agent harness).

Evaluation Suite : task collection measuring a specific capability/behavior (e.g., customer support suite testing refunds, cancellations, escalations).

Feb 2026: Lobster global boom spurred open‑source evaluation tools:

PinchBench (1200+ stars): benchmark for OpenClaw, emphasizes real‑scenario task simulation via complex tasks from actual work (md file task definition).

Claw Eval (500 stars): Peking University's Lobster capability evaluation task list (yaml file).

WildClawBench (500 stars): "wild environment" evaluation — test agents in real user OpenClaw scenarios, not curated sandboxes (Skill‑based task definition).