Cross-Layer Root Cause Analysis: How STAROps & Yaochi Agent Trace Alerts to Missing Indexes & Blocking Lua Scripts

When a frontend interface P95 latency spikes to 1800 ms, the alert propagates downstream: Redis connection counts fluctuate, RDS active sessions rise. The investigation narrows to the database layer, where generic observability dashboards are insufficient. STAROps excels at full-stack correlation analysis, collapsing the search space from multiple components to a few suspects. Once evidence points to the database, Yaochi Agent (alibabacloud-yaochi-agent) provides deep, metric-driven instance diagnosis and root cause analysis for Alibaba Cloud databases including RDS, PolarDB, Tair (Redis-compatible), MongoDB, Lindorm, AnalyticDB, ClickHouse, and SelectDB. Mounted as a Skill on a STAROps digital employee, Yaochi Agent extends the assistant with database-layer expertise, enabling a single evidence chain from application layer down to specific database commands.

Frontend Service Chain and Alert Propagation Path

The scenarios share an e-commerce application. The frontend receives user requests, routes them via frontend-proxy to business services (wishlist, promotion), which then call Redis (Tair) and RDS. When Redis single-thread blocking occurs, the downstream business services time out first; errors cascade up through wishlist/promotion → frontend-proxy → frontend, finally surfacing as a slow-interface alert.

Agent replies shown are final conclusions. In practice, the agent first displays analysis progress (querying metrics, analyzing slow commands). Single-instance diagnosis typically completes in minutes; cross-layer root cause analysis takes 5–10 minutes. If information is incomplete, the agent may ask follow-up questions.

Scenario 1: Active Session Spike — Root Cause Is a Missing Index (L3)

Question

Analyze the cause of abnormal active session increase on RDS instance rm-j6c3l32c730ti37ur during 2026-07-18T12:21:30+08:00 → 2026-07-18T12:38:30+08:00.

Agent Reply

Within the specified window, RDS active sessions surged from normal to peak, lasting ~17 minutes before subsiding. Analysis shows a large volume of queries concentrated on the inventory table, each containing a correlated subquery on the product_id column. Client source IP 10.0.20.210 opened 540 concurrent connections executing these queries. Because the inventory table lacks a product_id index, every query performed a full table scan, causing active session pile-up under high concurrency. Root cause: inventory table missing product_id index; 540 concurrent high-cost correlated subqueries saturated the RDS instance. Recommendation: add product_id index on inventory table.

Summary

This is an L3 diagnosis — the alert pointed directly at a single RDS instance without cross-layer propagation. Yaochi Agent's diagnostic path: confirm metric anomaly → analyze SQL distribution → trace client source (IP 10.0.20.210, 540 concurrent) → pinpoint missing index ( inventory.product_id). The precision from "active sessions rising" to "which table lacks which index" is sufficient to guide a fix directly in the RDS console.

Scenario 2: Alert in Frontend, Root Cause in Redis Lua Script (L4)

Question

Diagnose the root cause of a frontend slow-interface alert occurring 2026-08-10 10:31 to 10:50 and propose a solution.

Agent Reply

The digital employee first launches STAROps full-stack correlation analysis, using timeline comparison, topology tracing, and change-record cross-referencing to narrow the scope from frontend layer down to Redis layer. It then invokes Yaochi Agent for deep Redis diagnosis.

Deep Redis diagnosis reveals key metrics: CPU saturated at 98% while throughput dropped (QPS fell from 1200 to 180), indicating an operation blocking Redis's processing thread. Slow-command log analysis identifies the blocker: a Lua script containing heavy computation logic monopolized the Redis single thread and recurred twice during the alert window. Cascade propagation path: Redis single-thread freeze → promotion and wishlist services SocketTimeout → frontend-proxy returns timeout errors → frontend interface latency increases. Root cause: Managed Redis instance's single thread monopolized by a Lua slow command with heavy computation, recurring twice. Recommendation: refactor the Lua script to avoid heavy computation inside the script body; consider using pipeline to batch operations.

Summary

This is an L4 diagnosis — alert triggered at frontend layer, root cause in Redis layer, involving cross-layer propagation. Diagnostic path: frontend alert → STAROps full-stack correlation → scope narrowed to Redis → Yaochi Agent Redis metric analysis → slow-command log locates Lua script blocking → cascade propagation confirmed (Redis → microservices → frontend). Diagnostic precision reaches command type and behavioral characteristics, not just "Redis has a performance issue."

Scenario 3: Frontend Latency Spike, Root Cause in Redis EVAL Command (L4)

Question

Analyze the cause of a frontend response-time spike alert during 2026-08-10 06:19 to 06:25.

Agent Reply

Same investigation path as Scenario 2: STAROps full-stack correlation points to Redis layer; Yaochi Agent performs deep Redis diagnosis.

Redis instance metrics show CPU near saturation, QPS plummeting — a pattern similar to Scenario 2. Slow-command log analysis identifies the blocker: EVAL commands. These commands contain large amounts of inline logic, executing complex calculations directly inside Redis; the single thread is fully occupied during execution, forcing other commands to queue. Root cause: Redis single thread monopolized by EVAL slow commands; all commands queued. Recommendation: identify and block the client source issuing anomalous EVAL calls; optimize script logic to avoid large-scale loops.

Summary

Another L4 cross-layer RCA with the same "alert in frontend, root cause in Redis" propagation chain and nearly identical metric signatures. However, the underlying causes differ: Scenario 2 involved a resident business Lua script with heavy computation — fix by refactoring the script. Scenario 3 involved ad-hoc inline EVAL scripts (the Redis command for inline Lua) from an anomalous source — fix by blocking the offending client and moving computation out of Redis. Same symptoms, different remediation paths.

Comparison: Same Symptom, Different Causes

Both scenarios exhibit the same symptom and middleware, and the root mechanism is identical — "slow command monopolizes single thread" — yet the specific causes and remediation actions differ. This is the value of command-level diagnostic precision: it drives the conclusion to "which command, from where, and how to handle it" rather than merely stating "Redis has a performance problem."

Scope of Applicability

Yaochi Agent currently supports the following Alibaba Cloud database products: RDS, PolarDB, Tair (Redis-compatible), MongoDB, Lindorm, AnalyticDB, ClickHouse, SelectDB. Self-hosted and non-Alibaba Cloud databases are not supported.

When evidence is insufficient for a definitive conclusion, the agent annotates confidence levels and lists possible investigation directions instead of forcing a root cause. Therefore, before executing any fix, it is best to have a DBA or development lead review the diagnostic conclusion.

Yaochi Agent elevates database-layer diagnostic precision from "some middleware has a problem" to "exactly which command or index caused the issue," while STAROps stitches cross-layer evidence into a complete causal chain. Together, they let SREs and DBAs use a single toolset for end-to-end troubleshooting from alert to fix. For complex failures involving cross-layer propagation, this combination is worth trying.