Yidian Tianxia's Multi-Agent IT Ops Architecture: 87% Efficiency Gain & 5x Faster Root Cause Analysis

Yidian Tianxia, a global marketing technology company serving over 10,000 overseas enterprises, operates a high-scale, globally distributed advertising platform with millions of daily QPS, strong inter-service dependencies (SSP, DSP, ADX), cross-timezone 7×24 requirements, and extreme cost sensitivity. These challenges drove early adoption of AI for operations.

From Ad-Hoc AI Chat to Specialized Agents

Initially, engineers pasted logs into AI chat for Q&A-style analysis. By 2024, conversational AI assistants helped locate issues while humans executed commands. In H1 2024, widespread use of OpenClaw and similar Agents enabled direct command execution and proactive root-cause exploration, yielding 68–87% average efficiency gains across four high-frequency scenarios.

A concrete example: an Aerospike cluster scaled to 40/40 HPA and 80+ CA nodes appeared to be a cluster avalanche. An Agent pierced the symptom in 8 minutes, pinpointing a single-node connection-pool exhaustion; the human baseline required 40 minutes.

Four Problems with Individual Agent Adoption

Individual variance: Same Agent version and model produced wildly different results across engineers; teams asked top performers to “train our Agent.”

Guardrails unenforceable: Policies forbade Agents from executing write operations or accessing AK keys, but with everyone running personal Agents, enforcement was impossible.

Organization not reshaped: Humans remained the central drivers; AI merely assisted the same manual workflow.

Quality unreliable: Even top engineers saw Agent output quality drift over time due to Skill proliferation or model changes; production ops demand deterministic, stable quality.

Unified Multi-Agent Architecture on AgentCore

After evaluating vendor, open-source, and self-built options, Yidian Tianxia chose Alibaba Cloud AgentCore for its comprehensive coverage of operational needs. They built a closed-loop architecture with three layers:

1. Data Layer (Unified Data Foundation)

Integrates CloudMonitor 2.0 (metrics, traces, entities, topology, events), SLS (logs and analysis), CMDB (resources and topology), and internal systems via MCP (standardized invocation). All observability data converges into a single view for Agents.

2. Unified Intelligent Entry (Collaborative Execution)

AgentCore governs operational Agents with a Leader Agent at the center. The Leader handles intent recognition, scenario routing, task orchestration, multi-Agent collaboration, and result aggregation. It coordinates specialized Workers: Container-RCA Agent, Troubleshooting Agent, DB-Agent, STAROps Agent, etc. Operational Skills are centrally managed and security-scanned, exposed via AI Native entry points (Skill, CMS MCP Server, Inner MCP Server).

3. Agent Audit & Self-Evolution

Combines Agent observability and AgentLoop to record call chains, sessions, tokens, and execution trajectories. Identifies security risks and drives self-evolution through evaluation experiments. Memory and knowledge are stored in ContextDB.

In one sentence: Leader owns process and guardrails; Workers own domain diagnosis; isolated knowledge reduces crosstalk; quality gates ensure trustworthy answers.

Multi-Agent Ticket Processing Pipeline

Business issues enter via natural language, tickets, or IM bots (e.g., “build failed, log shows dependency download timeout”). Common types: databases (MySQL/PG/Redis/MongoDB/Aerospike/Cassandra), builds (logs/dependencies/pipelines), containers (K8s/Pod/node/network), Alibaba Cloud ops (common issue analysis/long-term inspection). The pipeline runs in five steps:

Business Problem Entry: No need to know which team to contact.

Leader Agent Triage: Identifies business domain, selects SOP, dispatches and collaborates, aggregates evidence, enforces boundaries (no overreach, no SOP skipping, no conclusions without evidence, no direct expert diagnosis).

Specialized Worker Agent Team: Database Agent, Build Agent, Container Agent, STAROps Agent each handle their domain. Every Worker combines “knowledge base + experience + Skill/MCP” — business knowledge, historical trajectories, success paths, BadCase recovery strategies, and tools like read-only DB queries, log search, K8s diagnostics, build platform APIs. AgentLoop records traces, builds datasets, evaluates quality, and regresses BadCases.

Quality Gate & External Reply: Leader decides if the conclusion meets release standards: SOP steps complete, evidence chain intact, conclusion consistent with evidence, permissions and risks acceptable.

Human Intervention: Passed conclusions output self-service answers (conclusion, evidence, suggestions, next steps); failed ones request more info, avoid evidence-free answers, and escalate non-deterministic overreach to humans.

AgentCore solves three engineering challenges in multi-Agent collaboration: inter-Agent authentication (credentials, expiry handling), heartbeat monitoring for variable task durations with re-dispatch on failure, and isolation between Workers plus long/short task communication.

STAROps & CloudMonitor 2.0: Automated Metric Onboarding

Unified data layer is foundational. With multi-cloud, multi-account, multi-region deployments, Agents must not fetch data piecemeal. Example: ECS metric onboarding. Traditional console or API approaches are labor-intensive and error-prone. Using STAROps, Yidian Tianxia achieved batch onboarding of specified metrics for 30+ customer accounts via intelligent conversation. New ECS instances with proper tags auto-register to managed Prometheus. Adding a new cloud account requires only a simple dialogue with STAROps to complete per-Region collection, multi-Region aggregation (Americas→Virginia, Southeast Asia→Singapore, Europe→Frankfurt), and unified delivery to Alibaba Cloud Singapore Region Prometheus. Agents then run automatically, initializing once and recurring per design, producing unified monitoring data for downstream Agent consumption.

Self-Evolution Flywheel: Turning Every Ops Run into Best Practice

AgentLoop’s experience flywheel runs six steps:

Observe Traces: Collect model, tool, error, recovery process, and final result.

Clean Trajectories: Denoise to standard trajectories; data volume ~4–6% of raw traces.

Mine Experience: Extract success paths, anti-patterns, parameter rules, recovery strategies.

Evaluation Gate: Verify via success rate, stability, and unit success cost; valid experiences enter store.

On-Demand Recall: Inject minimal experience keyed by goal, object, tool, progress, error state.

Better Execution: Fewer detours at critical nodes; new traces feed next cycle.

The store holds “how-to” methodology (Experience Store), not raw logs or business knowledge. It covers six real SRE scenarios: cost analysis, capacity inspection, gateway self-check, proactive notification, fault root-cause analysis, daily work. In reference experiments, accuracy rose from 7.1% to 36.1%, token usage dropped 6.8%, moving from “occasionally correct” to “stably production-ready”: higher average quality, raised floor, reduced variance, lower unit success cost.

Agent Observability & Security Audit

Operations cannot bypass Agent security. Beyond AgentCore’s pre-deployment Skill analysis, Yidian Tianxia integrated an Agent Security Center with continuous pre/in/post runtime analysis. Audit chain: application semantics + runtime facts → rule-based screening → semantic judgment → risk grading → investigation & verification, answering five questions: intent, actual action, boundary violation, impact scope, evidence location. Achieves pre-execution control, in-execution tracing, post-execution judgment — turning Agent risk from “invisible” to discoverable, sortable, investigable, verifiable.

Key principle: rules handle deterministic judgments; models handle context-dependent semantic judgment. LoongSuite-Pilot probes collect semantic facts (session, model, tool); LoongCollector eBPF collects behavioral facts (process, file, network). Logic rules match sensitive clues for deterministic verdicts; model semantic judgment combines task intent, session context, authorization boundaries, and execution results. Standardized audit facts enter unified audit pipeline, outputting typed, leveled, reasoned conclusions with hit fields and evidence locations, aggregated by subject, type, time, sorted by level, trend, impact scope, feeding anomaly behavior and entity investigations with original behavior record replay.

Three Hard-Won Lessons

Respect business process first. Don’t lead with technology. Ask: when can we solve the first problem? What capabilities are truly needed? Engineers excel at “how to implement”; the critical question is “what to implement.” Every Agent project must start from a concrete business pain point, clarify the scenario, then match the tech stack — wrong scenario makes even the strongest tool a castle in the air.

Prefer vendor solutions over DIY. Small teams lack capacity; mid-sized teams with dev resources often want “flexible control” via self-build. But early phases sink effort into foundations, loops, observability — delaying business value. Auth, audit, evaluation, orchestration are mature cloud products; self-build equals long-term cost, purchase equals speed. Save energy for business Agent and Skill design.

Continuous operations matter more than delivery. Agent result quality cannot stay stable forever; there’s a long tail of operations. An Agent is not a one-off deliverable but a “new hire” needing continuous training, evaluation, correction. Data drifts, models upgrade, business changes — without AgentLoop, Agents slowly “degrade.” Future ops engineers will become dedicated operators of specific expert Agents, ensuring stable, reliable output — making “operate Agent” a standing role.

AI-driven ops efficiency ≠ buying a tool. It equals Business Process × Digital Employees × Continuous Operations . Clarify the process first, let Agents truly embed into workflows instead of staying in individual hands; use a unified digital employee architecture (Leader Agent scheduling, specialized Worker diagnosis, quality gate fallback) to turn individual capability into organizational asset; then run AgentLoop’s evaluation–experiment–self-evolution loop continuously so every ops run becomes the next best practice. This is Yidian Tianxia’s complete path to IT operations organizational efficiency and transformation.