Building a Full-Agent Observability and Quality Evaluation System: From Data Collection to the Data Flywheel

Background: New Challenges in the Agent Era

As large‑model agents move from demo to production, traditional observability and quality‑assurance systems must be rebuilt. Four major challenges arise:

Fragmented application frameworks : LangChain, LlamaIndex, Dify, Spring AI, etc., each with different abstraction levels, call patterns, and context propagation, making unified instrumentation difficult.

Complex inference execution chains : RAG, Function Calling, Model Context Protocol (MCP), Agent‑to‑Agent (A2A) protocols create multi‑step, multi‑component workflows that exceed the coverage of classic APM tracing.

Expanded performance metrics : Beyond latency and throughput, agents require TTFT (time‑to‑first‑token), TPOT (time‑per‑output‑token), SSE streaming quality, and conversation turn counts, all directly affecting user experience.

Diverse data‑collection goals : Token consumption, text, image, audio, video, and other multimodal data demand sampling, compression, and sensitive‑content sanitization.

Full‑Link Observability Architecture: Three‑Layer Design

Access Layer – Multi‑form, Non‑intrusive, Standardized

High‑code agents (LangChain, LlamaIndex, AutoGen, Spring AI): deep SDK instrumentation.

Low‑code platforms (Dify, Langflow): platform‑level extension injection.

General agents (OpenClaw): runtime‑level unified collection via bytecode enhancement or eBPF.

Multi‑language support (Python/Node.js/Java/Go): bytecode or eBPF based non‑intrusive hooks.

Key design : Adopt the OpenTelemetry (OTEL) standard for ecosystem compatibility and release the LoongSuite open‑source SDK for GenAI semantic extensions, achieving framework‑agnostic instrumentation.

Compute & Storage Layer – Agent‑Native Data Processing

GENAI semantic alignment: map technical spans to semantic roles (llm_call, retriever_search, tool_execution) to bridge function calls and agent decisions.

Trace tail sampling: sample based on errors, slow requests, anomaly patterns to balance storage cost.

UModel entity topology: build a graph of LLM models, tools, knowledge bases, and agent instances, upgrading from raw trace to system‑level understanding.

Application Layer – Three‑Dimensional Analysis

Performance analysis : token consumption, LLM/tool latency, average inference rounds – used to optimise user experience.

Security audit : sensitive‑info leakage, over‑privileged tool calls, prompt injection – supports compliance and risk control.

Cost analysis : model‑call cost trends, session cost accounting – informs FinOps decisions.

Offline Evaluation System: Guarding Against Agent “Intelligence Degradation”

3.1 Evaluation Pipeline Design

Test set maintenance → Environment preparation → Test case execution → Evaluation execution → Evaluation result → Quality gate

Quality‑gate mechanisms :

Baseline comparison: new version vs. old version metric diff to detect regressions.

Quality checkpoints: core cases must pass 100 %; edge cases allow tolerance.

Gate rules: hard thresholds (block release) vs. soft thresholds (alert).

3.2 Test‑Set Construction: Three‑Element Model

Task input : question / multi‑turn context + scenario tags – requires exact recreation of dialogue state.

Runtime configuration : state types, parameters, mock services – any factor that may affect agent behaviour must be declared.

Ground truth : expected result, key evidence, allowed/forbidden tool calls – validates both what the agent did and what it did not do.

Maintenance can be spreadsheet‑based for quick entry or Git‑repo based for engineering‑grade version control. The repository follows a standardised directory layout (skill‑name/files, tests, eval.json, setup.yaml).

High‑quality test‑set traits :

Scenario coverage – aligns with real‑world task distribution, prioritising core scenarios.

Reproducible environment – each case binds an environment snapshot, avoiding live‑service dependencies.

Dynamic update – anonymised online cases flow back to enrich the test set.

3.3 Simulation Environment: Deterministic Guarantees

Static read‑only interface : document queries, data immutable – realised via local mirrors or versioned indexes.

Mock services : stateful/tool APIs – record real responses and replay fixed data.

Sandbox : run tools in isolated Docker containers with OverlayFS and network namespaces – prevents permanent file changes and cross‑task interference (e.g., /workspace supports 100 parallel tasks).

3.4 Evaluator – The “Judge” System

Human evaluation : expert review, customer feedback – high accuracy, high cost; suited for benchmark labeling and dispute arbitration.

Code evaluation : unit tests, string similarity, round count, token usage, tool call allow‑list – low cost, stable; ideal for deterministic tasks and CI gates.

LLM/Agent evaluation : scale scores, assertion scores, reference matching, pairwise comparison – flexible but more volatile; used for open‑ended tasks.

LLM Evaluation Methods

Scale scoring – multi‑dimensional 1‑5 rating for fine‑grained quality analysis.

Assertion scoring – Boolean assertion set, more objective than scales.

Reference scoring – semantic matching against ground truth (non‑literal).

Pairwise comparison – compare two outputs to eliminate absolute‑scale bias.

Volatility Mitigation

Apply N‑shot voting, manual calibration, and iterative optimisation to stabilise LLM‑based scores.

3.5 Agent Trace Evaluation: From Trace to Evaluatable Data

Raw traces are dense and exceed LLM context limits. A three‑stage processing pipeline (illustrated in the image) extracts decision logic and tool‑call relationships while discarding timing, token counts, and internal framework noise. The tool‑return payload is compressed to a ${response} placeholder preserving the call structure.

3.6 Skills Repository Evaluation Practice

Evaluation targets “Skill” (tool) level agents using a multi‑agent collaboration architecture:

Main agent : orchestrator that distributes tasks.

Task sub‑agent : executes concrete use‑case steps.

Eval sub‑agent : assesses the output.

This “agent evaluates agent” meta‑design ensures extensibility and consistency.

Online Evaluation System: Bridging Simulation and Reality

4.1 Limitations of Static Benchmarks

Dependency drift : third‑party API upgrades or rate‑limits cause online failures – mocks capture snapshots, but APIs evolve.

Long‑tail inputs : ambiguous, multi‑intent tasks lead agents astray – test sets are curated, production is open‑domain.

Conversation state accumulation : token overflow or memory conflicts degrade capability – offline tests are short‑session, online sessions accumulate state.

Coverage gaps : boundary, exception, combinatorial scenarios are rarely covered – combinatorial explosion leaves coverage < 1 %.

4.2 Two Pillars: Real‑Time Alerts & Badcase Mining

Online traffic → Anomaly detection → Clustering → Root‑cause classification → Sanitised feedback → Test‑set update → Offline validation → New version release

Real‑time alert strategy : monitor TTFT, TPOT, token consumption, tool‑call success rate, negative user feedback; set dynamic thresholds and trend detection.

Badcase mining workflow (illustrated in the image) extracts abnormal cases, sanitises them, feeds them back into the test set, and triggers offline verification before deployment.

4.3 Online Processing Engine: Technical Implementation

CPU + GPU hybrid: CPU aggregates traces and matches rules; GPU runs LLM evaluator inference and intent classification.

Scenario‑driven evaluation: automatic intent classification + dynamic tag system routes traces to appropriate evaluator collections.

Semantic deduplication: embedding‑based clustering reduces 10 000 traces to ~150 candidates, making manual review feasible.

Data Flywheel: The Perpetual Engine for Quality Improvement

5.1 Four‑Stage Loop

Full‑link observability → Trace evaluation → Iterative optimisation → Experiment evaluation → (back to full‑link observability)

Full‑link observability : deep trace reconstruction, performance/cost/security metrics – provides raw data fuel.

Trace evaluation : task completion, trace efficiency, abnormal case filtering – yields actionable evaluation results.

Iterative optimisation : derive optimisation direction from evaluation and trace, produce concrete fix suggestions.

Experiment evaluation : pre‑release quality gate, enrich test set with online data – validates effect and solidifies knowledge.

5.2 AIOps Agent Iteration Flywheel: Engineering Practice

In AIOps scenarios, the flywheel materialises as a complete engineering workflow (see image):

Chaosblade fault injection : simulate network latency, service crashes, resource exhaustion to verify agent resilience.

Online issue replay : replay historical online problems in a gray‑environment to confirm fixes.

Human‑machine collaboration boundary : automate massive trace processing, let humans handle root‑cause judgement and priority ranking.

5.3 Differentiated Evaluation for Three Task Types

Numeric/Time‑series (e.g., metric queries): coverage, point‑pass, Pearson, NRMSE – evaluated with code tests.

Toolchain/Structure (trace/log/event): expected tool calls, query rationality, tool‑intent match – evaluated with code + LLM.

Semantic/Answer quality (consultation, open questions): LLM judge + tool list verification + trace assistance – evaluated with LLM.

5.4 Three‑Stage Evolution Path

Cold start : manual expert‑driven sampling and review.

Automated regression : template‑driven automation builds repeatable regression capability.

Engineering‑level continuous evaluation : system‑driven pipeline – trace extraction → analysis/comparison → filtering → semantic deduplication → candidate selection.

Conclusion: Core Design Philosophy

Offline + online complement each other; offline provides determinism, reproducibility, and risk containment, while online offers real‑time, dynamic, adaptive handling of unknown risks.

Both feed Badcase feedback loops to close the quality‑improvement cycle.

Layered information processing filters raw trace → structured trajectory → evaluation result, preserving decision logic while discarding noise.

Evaluation is differentiated per task type: numeric precision, tool‑chain compliance, semantic reasonableness.

The data flywheel outperforms single‑point optimisation by continuously refining test‑set realism, evaluator calibration, and root‑cause pattern recognition.

Key Terminology Index

TTFT – Time To First Token.

TPOT – Time Per Output Token.

MCP – Model Context Protocol (Anthropic).

A2A – Agent‑to‑Agent protocol (Google).

OTEL – OpenTelemetry.

LoongSuite – GenAI semantic‑extension SDK.

UModel – Entity topology model for agent relationships.

SLS – Alibaba Cloud Log Service.

Chaosblade – Open‑source chaos‑engineering tool.

N‑shot Voting – Multiple LLM evaluations aggregated by voting.

Pairwise Comparison – Comparative scoring to remove absolute‑scale bias.

Ground Truth – Expected result or correct behaviour pattern for a test case.