How to Build a Scalable AI Evaluation Platform for Rapid Product Iteration

Background & Challenges

Rapid advances in large‑model AI have enabled four major use cases in Alibaba’s Taobao Flash Sale: digital‑human assistants, data‑analysis & decision products, multimodal content creation, and AI‑enhanced search. Testing these AI products is difficult because their behavior is dynamic, uncertain, and composed of long agent chains. Traditional functional testing cannot guarantee quality, leading to challenges such as fast‑changing architectures, loss of gold‑standard data, diverse versioning, and the need for both offline and online validation.

Evaluation System

The testing process is upgraded from a simple acceptance check to a quality‑engineered system that spans the entire AI product lifecycle. Core principles are:

Standard‑setting : product, design, engineering, and business stakeholders jointly define metrics covering business impact, correctness, performance, safety, and cost.

Dual‑track assurance : offline baseline tests are run in parallel with online effect monitoring.

Data reuse : annotations from development, acceptance, and production are consolidated into reusable gold‑standard datasets and automated regression cases.

Evaluation Dimensions

Business goals : conversion, retention, GMV, human‑task substitution.

Product effect : answer accuracy, helpfulness, tool selection, fidelity, logical consistency, hallucination.

Performance & experience : latency, multi‑turn flow, truncation, user satisfaction.

Safety & compliance : content safety, data privacy, regulatory compliance.

Service & cost : stability, inference cost, resource efficiency, operational complexity, ROI.

Testing Strategies

End‑to‑End vs. Layered Testing

End‑to‑End testing mirrors real user interaction, provides clear business‑level outcomes, and is ideal for version comparison, but it cannot pinpoint the failing module. Layered testing isolates intent recognition, tool planning, and retrieval, enabling precise debugging; however it requires extensive test‑case maintenance and can become brittle when architectures change.

Reference‑Based vs. Reference‑Free

Reference‑based tests use predefined correct answers and suit structured Q&A, extraction, or parameter‑validation scenarios. Reference‑free tests handle open‑ended generation, multi‑turn dialogue, or creative writing where no single answer exists.

Practical Implementation

For reference‑based scenarios, build a deterministic replay environment that records external tool inputs/outputs, timestamps, and context. During replay the recorded data are injected, guaranteeing repeatable execution of gold‑standard cases.

For reference‑free scenarios, adopt an LLM‑as‑judge approach: design a multi‑dimensional rubric (correctness, completeness, logic, safety) and use either a generic LLM (e.g., gpt‑4) or a fine‑tuned judge model. The judge can be augmented with retrieval and tool‑calling capabilities to fetch supporting evidence before scoring.

Combine model judges with rule‑based checks (format validation, blacklist enforcement) and a small human sampling set for calibration and bias mitigation.

Coverage & Efficiency

A three‑tier strategy selects test sets based on change scope and risk:

Small changes (prompt tweaks, ranking weight adjustments, UI copy updates): run a minimal smoke suite covering core flows and high‑frequency scenarios.

Medium changes (new tools, knowledge sources, agent strategy updates): run targeted end‑to‑end tests for affected business paths plus sampled high‑risk cases.

Major changes (model swaps, large‑scale workflow rewrites): execute full regression covering core, long‑tail, safety, and privilege scenarios, supplemented by adversarial samples.

Test cases are tagged with a taxonomy (business dimension, risk level, system link, tool involvement, deep‑thinking flag) that drives automatic selection. Example tag schema:

{
  "business_dim": "search",
  "risk_level": "high",
  "system_link": "tool_A->agent_B",
  "deep_thinking": true
}

Online Effect Evaluation

Data collection (user feedback, system logs) feeds a monitoring‑analysis‑optimization loop. Automated pipelines detect anomalies, trace root causes via link‑analysis tools, and trigger iterative improvements.

Platform Architecture

A modular “standardized process + plug‑in extension” platform was built over a year. It abstracts common evaluation capabilities as reusable services and supports multiple protocols (HSF, TPP, Whale) and data sources (Excel, ODPS, logs). The architecture separates workflow orchestration from concrete implementations, allowing rapid integration of new evaluation techniques (e.g., new metrics, judge models, or adversarial generators).

Key achievements (as of Sep 2025):

Supported >10 internal departments and >90 AI products.

Managed >1,000 test cases, 650 scenarios, and 67 judge templates.

Identified >200 issues with an 80 %+ resolution rate.

Executed >12,000 tasks processing >1.5 million data items with >95 % success and >85 % 24‑hour query resolution.

Future Directions

Multimodal evaluation : extend the platform to handle image, audio, and video AI products, integrating multimodal LLM judges and visual quality metrics.

Visual annotation workbench : provide a UI that renders agent chains and technical components visually, lowering the barrier for business users to create and review annotations.

Evaluation plugin marketplace : define a common plugin interface so teams can contribute custom safety rules, domain‑specific scoring models, or specialized agents, fostering an ecosystem of reusable evaluation capabilities.