Open-Source Foundations for AgenticOps: Standardized Collection, Unified Semantic Modeling & Reproducible Benchmarks

Standardized Data Collection: Bringing Runtime Facts Back Reliably

The article opens with the core challenge: moving AgenticOps from demo to stable production requires three layered capabilities — complete runtime facts, unified fault context, and reproducible capability verification. Alibaba Cloud's Observability team addresses these through three open-source pillars.

OBI: Zero-Code OpenTelemetry Signals via eBPF

OBI (OpenTelemetry eBPF Instrumentation) targets systems that cannot be recompiled or instrumented. Using kprobe/uprobe, it intercepts network requests inside the TLS boundary (pre-encryption/post-decryption, no keys needed), performs user-space semantic restoration (HTTP/protocol parsing, provider detection, large-packet reassembly), and emits standard OTel GenAI Spans. It auto-detects LLM protocols — OpenAI, Anthropic, Gemini, Qwen — capturing inputs, outputs, tool calls, MCP, RAG, and embeddings. A key use case is Agent Sandbox observability, where invasive instrumentation is impractical; OBI restores non-instrumentable scenarios into correlatable, comparable GenAI semantics.

Domestic Framework Contributions to OpenTelemetry

The team contributed extensive framework support, exemplified by Apache Dubbo. The contribution path: (1) discover collection issues in production, feed bugs upstream, fix iteratively, replace deprecated APIs; (2) co-design semantic conventions for RPC frameworks with the community, define dedicated Dubbo RPC metrics, extend from Trace to Metric, establish independent SemConv pages; (3) rebuild instrumentation per spec, fix field semantics, align with spec; (4) promote to stable after hardening. This 1–2 year process created a repeatable paradigm now extended to gateways, databases, schedulers, messaging, etc.

Semantic Conventions: Dual-Track Collaboration

GenAI SemConv defines what to collect, naming, units, and methods. The community demanded real-world validation before merging. Alibaba ran two tracks:

OTel upstream track: Initiated GenAI SemConv discussion → maintainers requested real-scene validation → incremental feedback of fields, tool & agent instrumentation.

LoongSuite validation track: Organized internal model-call & tool-execution semantics → large-scale validation across Alibaba Cloud, Alibaba Group, Ant Group → open-sourced LoongSuite GenAI SemConv.

LoongSuite enables rapid field/implementation validation in real scenes; mature proposals graduate to OTel for cross-vendor long-term maintenance.

Key Semantic Extensions

Entry/Step: Entry Span preserves original user input & final output (frameworks may mutate input). Step Span expresses one ReAct think-act-observe cycle; long traces fold by round for readable investigation steps.

Skill: Business-capability attribution between Agent and Tool, recording name, id, description, version. Enables downstream Skill-based analysis and optimization (as referenced in prior Skill-tuning practices).

Implementations cover AgentScope, LangChain, and coding agents (Claude Code, Codex, Hermes, OpenClaw, QwenPaw).

Request-Level OpenTelemetry Tracing for Inference Engines

Using SGLang as example, contributions enable end-to-end request tracing from Agent into the inference engine. Core mechanism (0→1): designed Request → Request Phase → Token three-level granularity, providing concurrent-dimension analysis views, solving context propagation across TP/DP/PP and PD disaggregation. Engineering hardening (1→sustainable): continuous completion of Gateway/Engine propagation, trace levels, unified phase metrics, Perfetto export, tiered CI testing. This yields per-token probability-level collection, pinpointing specific tokens, generation phases, cross-tenant resource interference. All contributed upstream and merged.

LoongSuite Pilot: Coding Agent Standardized Collection

One-stop collection for 20+ coding agents (Codex, Claude Code, Qoder, Cursor, OpenCode, DeepSeek, Harness, etc.), extending observability from service processes to closed-source agents, dev machines, and CI. Maps uniformly to GenAI SemConv, exports to any OTLP backend (SLS, CMS/ARMS, Langfuse, local/HTTP). Enables issue localization, token consumption analysis, eval-driven optimization, Skill tuning, agent tuning, and security auditing (detecting unauthorized data access or command execution).

Community Governance

Three-year stats: 432 contributions (259 PRs, 79.2% merged; 173 Issues). Distribution: Java 135, Go 147, eBPF 47, Python 10, SemConv 23, GenAI SemConv 12, community 22, website 16. Governance seats: Go maintainer (3/7), Java approver (1/11), GenAI SemConv approver (1/8) — 5 seats directly shaping rules and releases. Recent OBI member promoted to approver, adding another seat. Rules become cross-vendor reusable; users migrate without redoing collection or field mapping.

UModel Semantic Runtime: Organizing Fault Context Around Objects

Collected data alone doesn't guarantee Agent comprehension. UModel addresses the "context unreadable" wall by organizing scattered metrics, logs, traces, events around the fault object into a queryable, traceable, evidence-extensible investigation context.

Real Case: Checkout Service Success Rate Drop

OTel Demo Checkout Service: success rate fell from 99.6% to 92.1% (7.5 pp) in 3 minutes. Timeline: 09:58 start, 10:01 V2.4 release, 10:04 DB connections rise, 10:06 payment timeout. SRE intuition suspects the release, but temporal proximity ≠ causality. Agent must verify: which instances did the release affect? Which MySQL object maps to connection rise? Are timeout calls and success-rate drop on the same dependency path? Four key judgments: which object changed first? What evidence supports cause? How did anomaly propagate? What to verify next?

UModel Three-Layer Structure

Static Semantic Model: Defines object types (checkout, payment, MySQL, Pod), relationship types (calls, depends_on, owns, runs_on), data semantics (success rate, DB connections, request timeout, release), and rules/actions (release validation, DB pool, downstream deps). Without this, same object stays misaligned across systems.

Semantic Runtime: Dynamically compiles at task time, focusing only on current object, time window, and investigation-relevant facts. Three explicit conditions: focus object checkout-service, window 09:58–10:06, goal explain success-rate drop (not patrol). Irrelevant data excluded; only key data fed to Agent.

Task Context: Outputs object instances (Deployment/Pod/calls), relationship subgraph, state snapshots (99.6%→92.1%), key events (10:01/10:04/10:06), factual evidence (metrics/logs/requests/events), and suggested verification order (release → DB → downstream). Agent sees both the drop and the exact data slice it maps to.

Cross-Domain Investigation: Preserving Object, Time, Source

User request: "Investigate checkout-service 09:58–10:06 success rate drop 99.6%→92.1%." Spans release system, DB, call chain, monitoring logs. Agent executes read-only cross-domain investigation along three paths, incrementally filling evidence: 1. Change direction: which objects did 10:01 V2.4 affect? 2. Database direction: 10:04 connection rise details and related logs. 3. Downstream direction: relationship between 10:06 payment timeout and checkout success rate.

Agent must return entity_id, window, source_ref, plan — telling users what entity, time range, source, and which link in the overall chain this investigation serves. This preserves problem boundaries for human review or cross-Agent validation. Experiments show this improves diagnostic quality by 50–100% across models.

UModel Open Source: Object Graph, Query Service, Agent as Public Semantic Standard

Goal: multi-cloud unified semantic runtime for cross-product tracing with common language. Three parts: - Code repo: Apache-2.0, model/service/query/web/test — runnable, queryable, Agent-integrable asset. - Release & initiative: May 2026 open source, launching Universal Semantic Standard (USS) with XPeng, ZhiYu, ChuanJieTong, JLCPCB, ShenZhouShangLong. - Open interfaces: local service, umctl, MCP, OpenAPI, Web UI, SDK, examples — reusable entrypoints for varied developer habits.

Unified semantic runtime also fixes benchmark context: object relations, problems, query time/window, entrypoints — enabling fair cross-Agent score comparison on identical context.

BRiSE Benchmark: Making Capability Withstand Repeated Evaluation

Even with context ready, Agent correctness isn't guaranteed. Demo success or occasional production wins don't prove stable, reliable, SRE-trustworthy capability. Real system state, fault evidence, models, tools, Skills, prompts all vary. BRiSE fixes fault environments, tasks, and scoring rubrics so the same problem can be re-run by different teams under identical conditions for fair comparison.

Industry-Driven Task Coverage

Via STAROps engagements across industries, tasks must specify object, relations, time range, evidence sources, deliverables. Key insight: keyword hit ≠ correct answer. Financial log query requires explicit direction, source, destination, time range, field semantics to judge if anomalous requests concentrate on a service/path.

Repeatable Evaluation Pipeline with Three Roles

Benchmark App: Reproduces real objects & fault data. Behind "Checkout success rate dropped, why?" lies a complex simulation app with heavy fault injection.

OpsAgent (SUT): Evidence query & root-cause reasoning. Can be STAROps Agents, any Ops Agents, or general Agents. Confirms object/time, queries multi-source evidence; process & conclusion recorded.

BRiSE: Fixes environment, data, scoring rubric; unified scoring.

Separation enables independent checks: task validity/difficulty/discrimination; Agent behavior; result judgment. Fixed datasets & environments, recorded tool calls & evidence, unified scoring rules — every diagnosis reproducible, auditable, regression-testable.

Real-Production Scenario Coverage

Existing benchmarks (OpenRCA 1.0/2.0, SREGym, AIOpsLab) focus heavily on RCA — the "crown jewel" — but daily ops is mostly patrol & anomaly analysis, which current benchmarks barely cover. BRiSE splits by evidence complexity into three task sets, each re-runnable within fixed object/time: - Data Query (L2): 1,360 tasks; each records object, time range, available tools, expected result, query accuracy. Example: query specified app's 30-min request error rate; list top-5 CPU hosts with aggregation口径. - Patrol & Anomaly Analysis (L3): 418 tasks; 11 evidence sources, 19 tech domains, 1,576 observable signals; each records anomaly pattern, evidence matching rules, scoring. Example: identify K8s Pod restarts, resource pressure, replica anomalies; find new/surging anomaly patterns from log clusters. - Fault Diagnosis (L4): 764 cases; 150+ fault types, 20+ cloud products; each records fault env, ground truth, evidence chain, process scorer. Example: judge how traffic surge impacts downstream calls & alerts; distinguish code latency, dependency fault, release change.

Every level records object, time range, tools, expected output, scoring rules — reusable across models, harnesses, query tools.

Industry, Academic, Cross-Domain Evaluation Practices

Industry: Datadog (Bits AI Eval) continuously evaluates platform alerting, topology, logs, investigation; failures routed to product teams for retest/fix. ServiceNow, Elastic, Dynatrace connect offline datasets, online replay, human review, version gates. Typical chain: dataset → scorer → version regression; every version re-runs fixed suite, locates failures.

Academia: RCAEval, OpenRCA, OpenRCA 2.0 focus on root cause & evidence; AIOpsLab, ITBench, CloudOpsBench provide executable environments; SREGym, OpsEval 3.0 emphasize real-time & layered. Tasks evolve from offline tables to runnable microservices/cloud envs; scoring differentiates object, impact boundary, evidence, investigation process.

Cross-Domain Agents: SWE-bench checks test pass; AgentBench measures multi-env task completion; BFCL evaluates tool selection & call results; τ-bench checks state changes & rule constraints. All judge completion via environment state, test results, or independent rules — not Agent self-report — inspiring trajectory-based reasonableness checks.

RCA100/200 Released, Community Co-Construction

RCA datasets published, positive community feedback. RCA100/200 accessed from 36 countries/regions, 92 cities, ~2.1k unique visits. Fault categories span app code/runtime, microservice deps/self-hosted middleware, K8s/container platform, host/OS, cloud instances, network, LLM/Agent, traffic/security. BRiSE open-source driven by community via single release flow (submit asset → auto-verify → baseline eval → publish results → include in next version) with four participant roles: - BRiSE/UModel: provide specs, reference env, initial cases. - University teams: cover algorithms, models, harnesses, ablation. - External teams: submit baselines, drive ambiguous-case convergence. - Open-source developers: contribute envs, faults, tasks, evaluations.

Four planned tracks: fault handling (cautious — needs rigorous eval specs, reliable recovery verification, safety boundaries), cloud-native benchmark factory (backed by national key R&D, with CAS, Tsinghua, Fudan, Nankai), plus two others.

Conclusion

Three pillars: collection brings runtime facts reliably; UModel organizes investigation context around fault objects; BRiSE measures Agent capability via reproducible benchmarks. Together they form a community-sustained AgenticOps open-source foundation. The goal: reliably pull runtime facts, give Agents clear task-specific investigation context, and make AgenticOps capability withstand repeated, reproducible evaluation. Production-grade AgenticOps needs not just stronger models, but such a continuously built engineering foundation.