Building Trustworthy AIOps Evaluation Datasets: STAROps Root Cause Analysis Practices

Introduction

Alibaba Cloud STAROps is a production operations agent that connects multi-domain data, operations tools, and expert experience to move from reactive response to intelligent autonomy, ensuring 7×24 business continuity. This article shares the evaluation and tuning practices of STAROps, and is included in the open-source

https://aliyun.github.io/ai-agent-handbook/cases/chapter-25/aiops-agent-evaluation.html

.

Rapid Evolution of Operations Evaluation Sets

The article surveys over 30 operations-related evaluation resources plotted by year and task domain. Key trends:

Quantity : New evaluation sets appear annually, mostly within the last two years.

Coverage : Early sets focused on telemetry diagnosis data and a few executable environments; later sets added root cause and causal process evaluation, extending to network, database, and AI infrastructure.

Evaluation Method : Shift from offline log/metric data (only checking final answer) to executable environments where the agent performs tasks, assessing evidence and whether the problem is actually fixed.

Evaluating the execution process demands that data preserve process information and evidence for each step: what data the agent queried, why a hypothesis was rejected, whether the time window was correct, and whether the issue was fixed.

A Single Root Cause Analysis Process

A root cause analysis is abstracted into three parts:

Input : e.g., three alerts — order error, product query error, frontend exception.

Multi-round loop of "Agent judgment → tool forensics": each round the agent scopes the problem, proposes hypotheses, excludes them with evidence, and decides the next step. Tools query metrics, logs, traces, events, topology, and change records. Context (evidence gathered, hypotheses excluded, historical experience) links rounds.

Diagnostic conclusion : root cause entity, propagation chain, impact scope, and remediation suggestions.

Correct conclusion ≠ trustworthy process. Three evaluation dimensions:

Generality : Does the reasoning work across diverse failure scenarios? Evaluation set must cover enough scenarios.

Error localization : When wrong, distinguish tool failure from reasoning failure. Data must record each step: tool called, tool output, judgment made.

Execution efficiency : Check for useless tool calls or context pollution; requires full call trace preservation.

A subtle error is time window mismatch : tool calls succeed and reasoning is self-consistent, but data comes from the wrong time window. Detection requires the evaluation data to store the true fault time window for comparison.

Requirements for Evaluation Data

Three basic requirements, each backed by external research:

Broad coverage : A public root cause dataset covers 27 injection mechanisms and 500 instances on three systems, yet real environments have more failure types.

Repeatable execution : Same agent on same scenario run 10 times succeeded only 6 times due to model non-determinism and environment changes. Single-run, single-success retention is insufficient.

Proven validity : Injection ≠ usable data. One study kept only 1,430 of 9,000+ injections because others caused no perceivable anomaly. Causal chain must be verified before writing ground truth.

These map to construction practices: plan topics by capability domain and difficulty , ensure environment stability and data replayability, and verify signal capture and evidence extraction before writing answers.

Composition of an Operations Evaluation Record

Each record must enable full reconstruction of the analysis trajectory . The trajectory has five steps:

Goal : what is being tested (e.g., locate root cause from three alerts).

Observation : data the agent sees (metrics, logs, traces, events, topology, changes).

Tools : tools used to query observations.

Actions : every tool call and reasoning step (scoping, hypothesizing, excluding, next-step decisions).

Result : final conclusion (root cause entity, propagation chain, impact, remediation).

Trajectory ≠ answer. Saving only the answer allows only conclusion comparison; saving trajectory reveals where the agent erred. Key judgments, evidence, and actions must be explicit.

Two additional artifacts saved with the trajectory:

Supporting conditions : environment, tools, observation methods.

Validation basis : evaluation criteria.

They must be co-located because the evaluation set is reused across agent versions; otherwise environment/tool changes make trajectories incomparable, and shifting criteria make scores ambiguous.

Four operations-specific constraints:

Environment binding : fault must occur in real services, traffic, observability; data saved with its environment.

Time binding : evidence has retention limits; save time window and captured evidence immediately for later verification.

Complete causal chain : every causal step must have evidence; ground truth separates cause, propagation, impact.

Low-frequency high-risk : production faults cannot be repeatedly tested; use fault injection in evaluation environment to reproduce same mechanism repeatedly.

STAROps Evaluation Set Construction Process

A 12-step, 4-phase pipeline:

Design (Agent) : select scenarios from capability map gaps and online issue replay; design fault (what to inject, where, propagation chain, impact).

Injection, Verification & Recovery (Tools) : confirm clean baseline, inject fault, verify effect (signal capture), wait for alert, restore environment. Injection success ≠ fault effect; must verify signals.

Forensics & Answer Writing (Agent) : capture evidence during and after fault; organize into ground truth with seven fields (see below).

Check & Storage (Program + Human) : Program gate checks rule-based items (field completeness, query re-execution consistency); Admission review checks judgment items (causal chain coherence, candidate exclusion) — first by Agent, then human final decision.

Any step failure triggers rework. The pipeline ends when STAROps (the evaluated agent) runs diagnosis on the same input and its output is compared field-by-field with ground truth.

Four Core Problems in Continuous Construction

Topic selection : addressed by capability map planning and online issue replay.

Environment construction : addressed by on-demand environment expansion.

Validity assurance : addressed by fault-injection-based construction loop.

High-quality iteration : addressed by quality checks.

These are interdependent: unstable environment undermines validity; uncontrolled topic sources waste effort.

Agent Division of Labor in Construction

Four specialized agents:

ScenarioAgent (Topic): gap identification, change analysis, architecture reconstruction, scene replay.

EnvironmentAgent (Environment): topology building, business simulation, traffic injection, observability integration.

BuildAgent (Construction): fault injection, signal capture, evidence extraction, admission review.

SupervisorAgent (Supervision): quality checkup, failure detection, regression retest, difficulty escalation.

Human role: define goals, define acceptance, final review . Humans set priorities (which capability gaps first), standards (required fields, evidence depth), and make final go/no-go. Repetitive construction and verification are delegated to agents.

Building a trustworthy evaluation set is itself a complex agent task, stress-testing an organization's agent engineering maturity across four dimensions: individual agent capability, inter-agent collaboration, supporting infrastructure (standardized tools, reproducible environments, context/state persistence, long-task recovery, handoff protocols, evidence traceability), and governance/quality mechanisms (programmatic checks, evidence verification, independent review, human final approval).

Planning Scenarios by Capability Domain and Difficulty

Proactive coverage uses a capability map :

Horizontal axis (capability domains): Code & Change, Application Runtime, Database, Container, Middleware, Cloud Product, Host, Network.

Vertical axis (difficulty): Single-point fault, Propagation chain, Concurrent multi-fault, Alert storm. Difficulty reflects how much noise must be excluded to pinpoint root cause.

Each cell contains concrete scenarios. Example: Database column — single-point: slow query; propagation: lock wait causing downstream timeout; concurrent: release + rate limiting. Empty cells = uncovered gaps, prioritized by humans then built by agents.

Restoring Online Issues to Evaluation Sets

Second topic source: real online incidents . They may fill existing map cells or reveal new mechanisms requiring new cells. Critical order: preserve first, then restore . When an incident occurs, immediately save traces, call chains, and time window (evidence retention is short). Then three steps:

Mechanism restoration (Agent) : clarify causality and necessary conditions; rebuild as a scenario in evaluation environment.

Equivalent replay (Tools) : re-execute same mechanism; not just reproduce error, but make the same cause recur.

Rebuild verification (Program) : align online vs replay signals (order, timing). Mismatch → restart rebuild. Passed data enters BuildAgent for standard admission review, ensuring uniform seven-field format and evidence standards.

On-Demand Expansion of Evaluation Environments

Environment expansion triggered by coverage needs (map gaps, new mechanisms, isolation requirements), not mere scale. Human defines capability domain and isolation boundaries (to avoid cross-fault interference). Evolution:

Start: open-source microservice demo, script-driven single order chain, only application-layer single-point faults.

Each new fault domain adds injection means and observability (e.g., host faults need host-level injection and metrics; cloud product faults need cloud product observability/alerts).

Cross-cluster and concurrent faults required cloning a second cluster, upgrading traffic to steady-state + scenario-driven layers (steady traffic makes environment realistic; scenario traffic exercises specific propagation paths). Alert thresholds calibrated via historical replay.

Now multiple clusters and business chains.

Each expansion in five steps: Agent reads dependencies & existing env → Agent orchestrates tasks (template-based service deploy/cluster clone) → connect observability & dual-layer traffic, establish alerts, calibrate thresholds, solidify repeatable injection/recovery configs → programmatic baseline check (clean steady traffic, queryable observability) → output reproducible environment configuration code (topology, business, traffic, observability, injection/recovery configs). Baseline check is strict because baseline anomalies mix with fault signals; unqueryable observability hides faults. Observability is the top priority — without it, fault reality cannot be confirmed.

Fault Injection-Based Construction Loop

BuildAgent runs a four-step loop; any failure loops back:

Fault Injection : three paths — reuse existing injection configs, modify configuration (for config-induced faults), modify code (for code-level faults). Must prove fault actually occurred ; command success ≠ fault effect.

Signal Capture : six mandatory checks — alert source, environment changes, metric anomalies, call chain propagation, impact scope. Critical watch: noise interference — if another fault runs in same time window, data is discarded.

Evidence Extraction : collect injection point, root cause, propagation chain, impact scope across four dimensions; organize into seven-field ground truth : Root cause entity — specific resource/service for precise scoring. Fault type — verifiable mechanism, not surface symptom. Causal chain — separate cause, propagation, impact; enables pinpointing where agent mis-attributed. Discrimination points — why similar candidates were excluded. Alert anchor — actual triggered event and time window (catches time-window errors). Key numbers — with original query, time window, entity; re-executable. Numbers without queries cannot enter ground truth. Example: an answer claimed ~25% error rate; re-query over five windows returned zero errors. Also, query result limits and baseline/fault window length differences can mislead. Scoring rules — per-field independent scoring; an answer can score high on root cause but low on propagation chain.

Admission Review : Agent reviews first (re-fetches evidence independently to avoid confirmation bias), then human final decision. Passed data enters evaluation set.

Quality Checks, Scale, and Next Steps

Post-admission, continuous five checks (short-board principle — no averaging away local problems):

Quality : evidence completeness, seven fields present, every number has executable query.

Coverage : type/difficulty distribution vs capability map gaps.

Discrimination : can scores separate strong vs weak agents? When both approach ceiling, actively increase difficulty (swap simple for propagation/concurrent/alert-storm scenarios). If spread reappears, original data lacked discrimination. Score drop after difficulty increase reflects harder tasks, not capability regression.

Scale : minimum samples per cell, total volume for statistical confidence.

Freshness : primarily sampling re-run success ; data age only auxiliary. Base image, alert rule, or observability changes can break yesterday's data.

Handling: Invalid data → retire (prevents misattributing agent failures to data decay). Agent fix → retest on still-valid data (mixed invalid data would confound improvement measurement).

Discrimination case: original data curves for strong/weak agents converged near top; after difficulty increase (vertical dashed line), strong agent recovered, weak agent fell, spread restored.

External 2026 research aligns with five checks: PAIChecker (quality), CORE-Bench (discrimination), confidence-interval sample sizing (scale), MMBench-Live (freshness). STAROps undergoes same checks.

STAROps evaluation set scale (as of September) : 1,400+ total entries (including in-progress and retired). 838 validated root cause analysis cases covering 100+ fault types — grew from 58 in March to >10× in six months. Plus 500+ data retrieval cases to evaluate agent's data-fetching accuracy.

Three reasons to build proprietary sets despite public availability:

Service topology and alert rules are proprietary → topics must map to own capability domains/difficulties.

New online mechanisms absent from public sets → must replay online issues.

Environment changes demand instant re-run capability → achieved via environment config code and periodic sampling re-runs.

Cost Investment Structure

Three-phase investment schematic (not measured curves):

Startup : human-heavy; few scenarios, one environment, minimal model calls.

Environment & Judgment Completion : environment, traffic, observability, alerts built out — environment spend catches up to human; judgment criteria refined via trial runs — human spend peaks.

Reuse of Established Paths : environments reproducible, construction steps and checks reusable; human shifts from per-case building to standard maintenance, exception handling, quality audit — repetitive human cost drops; model calls rise with volume and surpass human.

Eventually environment becomes largest cost (clusters, traffic, observability running continuously). Agent adoption does not necessarily reduce total cost — environment ops, model calls, quality audits persist; it eliminates repetitive manual work.

Public research confirms: 2026 automated pipeline ~$40 and 1-2 hours per accepted task, human does quality review; another 2026 agent pipeline with expert calibration of criteria/scoring. Both show automation with human oversight; cost structures must be measured per organization.

Open Source and Co-construction

Three directions:

Open source : gradually release built evaluation sets (RCA100, RCA200 released; more data and evaluation methods coming).

Co-construction : more teams contribute real environments, real scenarios, domain expertise for future construction.

Reconstruction : take public scenario descriptions, rebuild in real environments into full evaluation sets.

Co-construction and reconstruction are directions without public timelines.

Next automation step: evaluation-side agent proactively detects gaps from map blanks, sustained near-ceiling scores, and new online issues; triggers next construction batch, auto-retests after capability updates, then escalates difficulty. Focus: connect gap detection to construction tasks, verify new data restores discrimination via retest results.

Summary

This methodology changed four things:

Data rerunnable : environment, mechanism, evidence, answer, and criteria saved together.

Answers evidence-backed : every step has evidence, re-queryable.

Clear division of labor : humans set goals and final review; agents build and review (review re-fetches evidence independently).

Evaluation set evolves by capability gaps : next batch driven by map gaps, evaluation results, new online mechanisms.

Supported by three principles: ground truth only from evidence; review independence from evidence independence; evaluation set itself continuously evaluated and evolved . Beyond admission review, track three metrics: post-release error rate in approved data, maintenance of strong/weak score spread, data lifetime from creation to invalidation.

Building evaluation sets tests an organization's AI Native maturity . If data creation — from problem discovery, environment setup, evidence gathering, answer compilation, to quality checks — relies on manual stitching, even many agents may not constitute an AI Native workflow. The near-AI-Native state: humans define goals and standards; agents autonomously decompose and collaborate; infrastructure provides environments and tools; system auto-validates; humans handle only exceptions and final arbitration .