AgenticOps Open Source: Standardized Collection, Semantic Modeling & Reproducible Benchmarks

Three-Layer Open-Source Foundation for AgenticOps

As large models drive Agent adoption across domains, moving AgenticOps into production requires more than stronger models—it demands a sustainable engineering foundation. Alibaba Cloud's observability team identifies three progressive challenges: reliably collecting runtime facts, unifying fault context for Agent understanding, and reproducibly verifying Agent capabilities. These map to three open-source pillars—LoongSuite/OBI for standardized collection, UModel for semantic runtime context, and BRiSE for benchmark evaluation—each developed through community collaboration.

Standardized Collection: LoongSuite and OBI

Five Collection Paths Converging on OpenTelemetry

The collection stack covers server-side, infrastructure, and client-side Agents, all emitting standard OTel semantic data via OTLP to any compatible backend. Three main instrumentation strategies apply under different conditions:

Compile-time instrumentation for rebuildable applications.

eBPF-based instrumentation (OBI) for legacy systems that cannot be recompiled.

Runtime instrumentation for framework-rich environments needing in-app semantics.

Choice depends on application constraints, semantic depth required, and runtime permissions.

OBI: Zero-Code, Zero-SDK, Zero-Restart Observability

OBI (OpenTelemetry eBPF Instrumentation) uses kprobe/uprobe to intercept network requests at the TLS boundary—capturing plaintext before encryption or after decryption without keys. It performs user-space semantic reconstruction (HTTP/protocol parsing, provider detection, large-packet reassembly) and emits standard OTel GenAI Spans. In Agent scenarios, OBI automatically identifies calls to OpenAI, Anthropic, Gemini, Qwen, and other LLM protocols, capturing inputs, outputs, tool calls, MCP invocations, RAG, and Embedding operations. A key use case is Agent Sandbox environments where invasive instrumentation is impractical; Alibaba Cloud's sandbox observability is built on OBI.

Domestic Framework Contributions to OpenTelemetry

Beyond eBPF, the team contributed extensive framework support for Java and other languages. Using Apache Dubbo as a case study, they followed a year-to-two-year contribution path: discover collection gaps in production, report bugs, iteratively fix and align with deprecated APIs, co-design RPC-specific semantic conventions with the community (adding Dubbo RPC metrics, extending from Trace to Metric, establishing independent SemConv pages), refactor instrumentation to match specs, and graduate to stable support. This paradigm now extends to RPC, gateways, database access, task scheduling, messaging, and more.

Dual-Track Semantic Convention Collaboration

GenAI semantic conventions are core to OpenTelemetry. The community required real-world validation before accepting specs, so Alibaba Cloud ran two parallel tracks:

Upstream OTel track: Initiate GenAI SemConv discussions → maintainers request real-scene validation → incrementally contribute fields, tool and Agent instrumentation.

LoongSuite validation track: Consolidate internal model-call and tool-execution semantics → large-scale validation across Alibaba Cloud, Alibaba Group, and Ant Group → open-source LoongSuite GenAI SemConv.

LoongSuite handles rapid field/implementation validation; upstream handles cross-vendor long-term maintenance. Key extensions include Entry/Step spans (Entry preserves original user input vs. framework-modified input; Step represents each ReAct thought-action-observation cycle, folding long traces by round) and Skill spans (business-function attribution between Agent and Tool, recording name, id, description, version—enabling downstream Skill tuning). These work across AgentScope, LangChain, Claude Code, Codex, Hermes Agent, OpenClaw, QwenPaw, and other Coding Agents.

Request-Level Tracing for Inference Engines (SGLang)

To trace Agent requests through the inference engine, the team contributed to SGLang: (1) Core mechanism—designed Request → Request Phase → Token three-level granularity, providing concurrent-dimension analysis views and solving context propagation across TP/DP/PP and PD disaggregated nodes. (2) Engineering hardening—continuous fixes for Gateway/Engine propagation, trace levels, unified phase metrics, Perfetto export, and tiered CI testing. This enables per-token probability observation, pinpointing slow/error generation to specific tokens, phases, and cross-tenant interference. All contributions are merged upstream.

LoongSuite Pilot: Coding Agent Standardized Collection

LoongSuite Pilot targets Coding Agents (20+ supported: Codex, Claude Code, Qoder, Cursor, OpenCode, DeepSeek, Harness, etc.), collecting full execution traces from closed-source Agents, dev machines, and CI. Data maps to GenAI SemConv and ships to any observability backend (SLS, CMS/ARMS, Langfuse/OTLP, local/HTTP). Enables issue localization, token consumption analysis, evaluation/optimization, Skill tuning, Agent tuning, and security auditing (detecting unauthorized data access or command execution).

Community Governance and Contribution Metrics

Over three years: 432 contributions (259 PRs, 79.2% merged; 173 Issues) across Java (135), Go (147), eBPF (47), Python (10), SemConv (23), GenAI SemConv (12), community (22), website (16). Governance seats: Go maintainer (3 of 7), Java approver (1 of 11), GenAI SemConv approver (1 of 8)—5 seats directly shaping rules and releases. Rules entering OTel become cross-vendor reusable, eliminating re-instrumentation on migration. Recently an OBI member became an approver, adding another seat.

UModel: Semantic Runtime Organizing Fault Context Around Objects

The Context Gap: Data ≠ Understanding

Collected data alone doesn't let Agents diagnose reliably. LLMs can read error logs but lack alignment on underlying objects, relationships, temporal dynamics, and data provenance—leading to plausible but untrustworthy hallucinations. UModel addresses this by organizing scattered metrics, logs, traces, and events around the current fault object into a queryable, traceable, evidence-extensible investigation context.

Real-World Case: Checkout Service Success Rate Drop

Using OTel Demo's Checkout Service: success rate fell from 99.6% to 92.1% (7.5 pp drop) in 3 minutes. Timeline: 09:58 anomaly start, 10:01 V2.4 release, 10:04 DB connection rise, 10:06 payment timeout. SRE intuition suspects the release, but temporal proximity ≠ causation. Agent must verify: which instances did the release affect? Which MySQL object maps to connection changes? Are failing payment calls and Checkout degradation on the same dependency path? All facts must share one space and time window to judge correlation.

Three-Layer Architecture

Static Semantic Model: Defines object types (checkout, payment, MySQL, Pod), relationship types (calls, depends_on, owns, runs_on), data semantics (success rate, DB connections, request timeout, release), and rules/actions (release validation, DB pool, downstream dependency). Without this, the same object stays mismatched across systems.

Semantic Runtime: Dynamically compiles at task time, focusing only on the target object (checkout-service), time window (09:58–10:06), and goal (explain success-rate drop—not a patrol). Irrelevant data excluded; only investigation-critical facts fed to Agent.

Task Context: Outputs object instances (Deployment/Pod/calls), relationship subgraph, state snapshots (99.6%→92.1%), key events (10:01/10:04/10:06), factual evidence (metrics/logs/requests/events), and suggested verification order (release → DB → downstream).

Cross-Domain Investigation with Traceability

User query: "Investigate checkout-service 09:58–10:06 success rate drop from 99.6% to 92.1%." Agent executes read-only cross-domain investigation along three paths, each requiring entity_id, window, source_ref, plan fields so the investigator (human or another Agent) knows what object, time range, data source, and investigation step each finding serves. Paths: (1) Change direction—what objects V2.4 affected; (2) Database direction—10:04 connection rise details and logs; (3) Downstream direction—10:06 payment timeout vs. checkout success rate relationship. Experiments show this structured traceability improves diagnostic quality by 50–100% across different models .

UModel Open Source: Object Graph, Query Service, Agent Standard

Released May 2026 under Apache-2.0: runnable model/service/query/Web/test code—not just diagrams. Launched Universal Semantic Standard (USS) initiative with XPeng, ZhiYu Tech, Changer, JLCPCB, Shenzhou Long. Interfaces: local service, umctl, MCP, OpenAPI, Web UI, SDK, examples. Unified semantic runtime also fixes evaluation context: object relations, problems, query times/windows, entry points—enabling fair cross-Agent benchmark comparison.

BRiSE: Reproducible Benchmark for Agent Capability Evaluation

Why Benchmarks? Demo Success ≠ Production Reliability

Agents may succeed in demos or occasional production wins, but real systems change constantly—fault evidence varies, models/tools/Skills/prompts evolve. Without fixed conditions, improvement cannot be measured. BRiSE treats this as experimental design: freeze fault environments, tasks, and scoring rubrics so the same problem can be rerun by different teams under identical conditions for fair comparison.

Industry-Driven Task Coverage

Via STAROps engagements across industries, the team learned that keyword search ≠ correct answer. Financial log analysis, for example, requires explicit direction, source, destination, time range, and field semantics to determine if anomalies concentrate on a service/path. Every evaluation task must specify object, relations, time range, evidence sources, and expected deliverable.

Three-Role Evaluation Pipeline

Benchmark Application: Reproduces real objects and fault data (complex simulated app + massive fault injection behind "Checkout success rate dropped, why?").

OpsAgent (System Under Test): STAROps Agents, any Ops Agents, or general Agents—confirm object/time, query multi-source evidence, record investigation process and conclusion.

BRiSE: Freezes environment, data, and scoring rules; unifies scoring.

Separation enables independent validation: task validity/difficulty/discrimination, Agent behavior, result judgment. Fixed datasets/environments, recorded tool calls/evidence, unified scoring—every diagnosis becomes reproducible, auditable, and regression-testable.

Three-Tier Task Suite Reflecting Real Ops Work

Existing benchmarks (OpenRCA, SREGym, AIOpsLab) focus heavily on root-cause analysis (RCA), but daily ops is dominated by patrol and anomaly analysis. BRiSE builds three tiers by evidence complexity:

Data Query (L2): 1,360 tasks; each records object, time range, available tools, expected result, query accuracy. Example: query specified app's 30-min request error rate; list top-5 hosts by CPU with aggregation method.

Patrol & Anomaly Analysis (L3): 418 tasks covering 11 evidence sources, 19 technical domains, 1,576 observable signals; each records anomaly pattern, evidence matching rules, scoring method. Example: identify K8s Pod restarts, resource pressure, replica anomalies; find new/surging anomaly patterns from log clusters.

Fault Diagnosis (L4): 764 cases covering 150+ fault types, 20+ cloud products; each records fault environment, Ground Truth, evidence chain, process scorer. Example: judge how traffic surge impacts downstream calls/alerts; distinguish code latency, dependency failure, release change.

All tiers record object, time range, tools, expected output, scoring rules—reusable across models, harnesses, query tools.

Learning from Industry, Academia, and Cross-Domain Agents

Industry: Datadog (Bits AI Eval) continuously evaluates alerting, topology, logs, investigation; failures routed to product teams for retest/fix. ServiceNow, Elastic, Dynatrace connect offline datasets, online replay, human review, version gates. Typical pipeline: dataset → scorer → version regression.

Academia: RCAEval, OpenRCA 2.0 focus on root cause/evidence; AIOpsLab, ITBench, CloudOpsBench provide executable environments; SREGym, OpsEval 3.0 emphasize real-time and tiered evaluation. Tasks evolve from offline tables to runnable microservices/cloud; scoring differentiates object, impact boundary, evidence, investigation process.

Cross-Domain Agents: SWE-bench checks test pass; AgentBench measures multi-env task completion; BFCL evaluates tool selection/call results; τ-bench checks state changes and rule constraints. All judge completion via environment state, test results, or independent rules—not Agent self-report—inspiring trajectory-based verification.

RCA100/200 Released; Community Co-Construction

RCA100/200 datasets published: access from 36 countries/regions, 92 cities, ~2.1k unique visits. Fault categories span app code/runtime, microservice dependencies/self-hosted middleware, K8s/container platform, host/OS, cloud product instances, network, LLM/Agent, traffic/security. BRiSE open-source governance follows a unified release flow (submit asset → auto-verify → baseline eval → publish results → include in next version) with four participant types: BRiSE/UModel (specs, reference env, initial cases), university teams (algorithms, models, harnesses, ablation), external teams (baselines, ambiguity resolution), open-source developers (env, faults, tasks, eval). Four planned Tracks divide work; Fault Remediation Track requires rigorous eval specs to verify safe, reliable recovery. Cloud-Native Benchmark Factory, under a national key R&D program led by Alibaba Cloud with CAS, Tsinghua, Fudan, Nankai, targets auto benchmark construction across dev/test/ops lifecycle.

Conclusion

The three pillars—collection (LoongSuite/OBI) bringing runtime facts reliably, UModel organizing investigation context around fault objects, BRiSE measuring Agent capability via reproducible benchmarks—form a community-sustained AgenticOps open-source foundation. The goal: make facts collectible, faults readable, capabilities verifiable in real environments, moving intelligent operations from demo to stable production.