How 1688 Built a Self-Correcting Agent Evaluation System with Dual-Track Testing

1. Agent Evaluation Cannot Only Check Answers: From Real Tasks to Dual-Track Evaluation

Traditional software testing, foundation-model evaluation, and business Agent evaluation solve different problems. Business Agent evaluation targets the system composed of model, prompt, tools, data, environment, and interaction , answering whether tasks complete stably, where root causes lie, and whether a version can be released. Engineering requires collaboration across sample analysis, evaluation engineering, Judge engineering, and Agent engineering.

Real-World Scenario: A B2B E-Commerce Task with Six Potential Failure Points

A real task: user asks 1688 to find sockets compliant with Thai standards. The task passes through six stages: understanding, planning, tool use, data retrieval, decision, and delivery . The Agent must understand that "Thai standard ≠ universal hole", set Thai standard as a hard constraint, execute it during product search and attribute filtering, verify spec evidence, and tighten criteria based on user corrections. In a bad case, the system returned products but still mixed in "universal holes", and multi-turn correction did not converge.

Therefore, Agent evaluation cannot only look at the final answer . The authors decompose real tasks into Task × Trial × Trace × Outcome :

Task : clearly define input, constraints, and success criteria.

Trial : repeat execution for the same query; check both pass@k (at least one success) and pass^k (stable success across runs).

Trace : retain conversation, tool calls, and intermediate states.

Outcome : verify final state and deliverables in the real environment.

Deterministic facts and hard thresholds are handled by rules/code; semantic intent and evidence understanding are delegated to LLM Judge; anomalies enter evaluation self-check.

Offline Golden Sets vs. Online Logs: Complementary, Not Mutually Exclusive

First-version offline sets are built by domain experts: controllable conditions, comparable versions, good for regression and release gating, and easy to accumulate bad cases. Problems: inputs too regular, may not cover real distribution, risk of leakage, contradictions, and mapping misalignment. Online logs expose high-frequency tasks, system non-responses, quota/rate limits, multi-turn corrections, and new requirements, but suffer from missing logs, mixed versions, and lack of standard intent — requiring intent recognition, filtering, and making them evaluatable. The two are not either/or: fixed sets answer "did the change regress?", online logs answer "what did users actually encounter?" .

Online Log Sampling Results (June)

878 user sessions, 946 intent units, 288 valid sampled gradings.

Overall average score: 64.2; full completion rate only 42.4%.

Unsatisfied cases: Agent no-response 38%, points/quota anomalies 22.2%, missing key delivery content 22.2%, system execution errors 9.5%.

71.6% of issues originated from system and infrastructure , model capability being only one layer.

Closing the Loop: From Report to Automated Repair

Early process: manual trigger after release → serial execution → AI report → manual calibration and release; reports did not automatically trigger repair actions . Recent improvements chain version/timer log trigger → evidence attribution → issue routing → fix adoption → automatic regression → release gating .

Panoramic Architecture: Shared Contracts Across Two Entrances

Offline golden sets and online real logs are two entrances, but downstream they share the same Task, Trace, Rubric, Judge, attribution tags, and report contract . Standard SOP: build task → environment execution → collect evidence → hybrid judgment → attribution improvement → regression gating. Four engineering gates remain: executable standards, verifiable evidence, trustworthy Judge, non-regressive loop .

Dual Thresholds: Separating "Complete" from "Excellent"

For vertical e-commerce tasks, release gating uses two independent thresholds:

Execution completeness ≥ 90% : checks mandatory steps, key tools, task interruption.

End-to-end result score ≥ 80 : checks delivery completeness, direct usability, data/computation/logic trustworthiness.

This blocks "results look beautiful but key steps never executed" false passes .

Rubric Engineering: Turning Business Consensus into Machine-Executable Contracts

Rubric difficulty is not length but transforming business consensus into machine-executable contracts . "Reasonable recommendation, complete report" lacks check fields, failure boundaries, evidence requirements — same Judge easily improvises. Executable Rubric must specify check items, pass conditions, evidence sources, severity, and gating , e.g., recommended product count, country-standard attributes, trace original text or tool returns, and hard-constraint failure = immediate fail.

Fixed Rubric for Deterministic Long-Chain Tasks

Procurement, order tracking, replenishment, business analysis have clear SOPs, checked like "flight checklists": parse intent → invoke capability → retrieve data → compute analysis → user confirm → structured delivery. Same Trace reused for execution completeness, output completeness, and content validity. Fixed is not the answer, but the business facts that must occur .

Dynamic Instance-Level Rubric for Open-Ended Tasks

Example "summer ski jacket": category deviation and key info fabrication are red lines; lightweight, breathable, summer suitability, wind/water resistance matching target scenario are important; clear buying advice and trade-offs are bonuses. Dynamic Rubric generates "what to evaluate" per input, but "how to aggregate" remains unified rules .

Multimodal Tasks: Per-Item Gating to Prevent Averaging Out Critical Failures

A garment image with 9 criteria: 7 correct → 77.8% accuracy. Object counting, recognition/action, scene gist may be perfect, but object position can be 0/1 . If positional relationship is a key capability, must gate separately , cannot let other correct items average out this failure.

Judge Trustworthiness: Every Conclusion Must Pierce to Raw Evidence

Output structure: Observation → Analysis → Conclusion : cite trace original text, tool status, or delivery fields → explain why satisfies/violates Rubric → output Pass/Fail and failure tags. LLM handles "understanding evidence", rule engine handles "calculating scores" — cannot let model be both referee and scorer.

2. Evaluation Systems Also Make Mistakes: Moving Judge from Auto-Scoring to Continuous Accuracy Improvement

Machine grading is not the endpoint. After business confirms real issues, fix Agent's skill, tools, data, or system; if business deems evaluation misjudgment, return to Rubric, Judge, rules, dataset for review, replay historical cases. Business feedback adds semantic and behavioral boundaries; weak evidence, template judgments become new evidence thresholds; new scenarios add Rubric and regression cases. Business problems fix Agent, evaluation problems fix standards .

Large-Scale Audit Reveals Systemic Judge Issues

Scanned 900 Verdict records, 260 files, 10,026 check items :

56.7% of Fail items had weak-evidence problems.

46.4% of Pass items had perfunctory judgments.

4.7% of same-case scores differed by ≥30 points.

1.7% showed score-deduction inconsistency.

Root causes not only model strength: single prompt judging too many items causes attention decay, loose evidence format lets weak anchors pass, model both judging and scoring lacks deterministic fallback .

Seven Deterministic Gates at Runtime (Not in Prompt)

Encode "discipline" into runtime:

Evidence strength gate

Counter-evidence principle gate

Template detection gate

Over-skipping gate

Score recalculation gate

Variance adjudication gate

Numerical reconciliation gate

Input side: micro-batching and order randomization. Output side: forced evidence syllogism and per-item retry. What rules or Python reconciliation can solve, don't ask the LLM again .

Review Process: Targeted Re-judgment, Not Unlimited Re-runs

Flow: Worker initial review → single recheck → batch inspection → convergence gating. Single anomaly: targeted re-judge only same-tag subset; batch anomaly: whole batch re-judged at most once ; still uncertain cases go to small-sample human arbitration. Deployed highlights: per-item isolation, original-text evidence, score recalculation, consistency audit. Automatic review, limited re-judgment, cross-round memory and experience reuse still in progress.

Dual Entrances Converge to Shared Contracts

Offline evaluation uses versioned golden sets + historical bad cases, guards release via repeated runs, version comparison, key-slice attribution. Online log evaluation discovers real distribution, abnormal chains, high-frequency needs, unknown issues. Online issues after desensitization and stratification flow back into offline golden sets; fixes regressed, then continue online observation. Both sides must share Task, Rubric, Judge, attribution tags, and report contract .

Dual Thresholds in Practice: Counter-Intuitive but Necessary

Example: a channel hot-product query strategy scored 94.41 end-to-end (>80) but execution completeness only 89.29% (<90) → still not released . For B2B long-chain tasks, high score cannot compensate for missing key steps .

Scores Alone Insufficient: Must Reach Root Cause

One full evaluation covered 7,170 sessions, 3,206 problematic sessions, 12,354 failure details ; 44.7% sessions contained failures. Failures categorized into 10 classes : system execution anomalies, output completeness defects, result analysis errors, task planning omissions, tool call errors, inappropriate interaction strategies, data acquisition defects. Tags must reach "failure detail" granularity ; one session can have multiple issue types.

Evaluation as "Problem Translator" for Downstream Applications

Title rewrite exceeding "30 Chinese chars / 60 bytes" → improvement lands on output post-length validation and automatic rework on violation.

Main image generation first round didn't absorb reference image → extract subject, tone, composition at entrance and inject as constraints.

Procurement product search treated "Thai standard" as universal hole, four correction rounds didn't converge → turn country certification into hard filter, tighten query with negative feedback.

Without retest results, don't claim "improved by X%", instead provide verification metrics and next-round experiment design.

Three Evolutionary Phases

Phase 1: Human-driven collaboration — humans pick cases, invoke capabilities, review, decide fixes.

Phase 2: Automated chaining — version changes or cron jobs auto-trigger environment execution, grading, attribution, reporting.

Phase 3: Evaluation-driven self-iteration (in progress) — results auto-consumed by downstream, fixes auto-regressed, continuous governance of evaluator's own bias.

What changes: initiation authority, orchestration method, result consumers.

Common Pitfalls (Mostly Not About Judge Model Selection)

Only looking at final answer.

Standards are vague consensus, no thresholds, evidence, or gates.

Evaluation sets too clean or self-contaminated.

Letting Judge compute scores itself.

Judging too many items at once → position decay and template laziness.

Auto-reports chase speed without recheck.

Only total score, no attribution.

Three must-dos first: define "complete" clearly, enforce evidence, evaluate offline and online simultaneously.

3. From "Runnable" to "Trustworthy": Making Evaluation Truly Enter Agent Iteration Loop

Three concluding takeaways:

Evaluation object is not the answer, but intent, trace, delivery, and behavioral boundaries.

Judge is not the endpoint; it must be constrained by evidence, rule gates, and review loops.

Score is not value; only when evaluation can attribute, flow back, and verify fixes does it truly enter the business loop.

The evaluation system itself also needs to be evaluated.