Tracing Business Alerts to Database Root Causes with STAROps and Yaoci Agent

Incident Overview

Frontend P95 latency spiked to 1800 ms. Tracing the call chain showed fluctuating Redis connections and a rise in active sessions on an RDS instance, narrowing the suspicion to the database tier.

STAROps Full‑stack Correlation

STAROps aggregates metrics, logs, and traces across application, middleware, and infrastructure layers to shrink the investigation scope to a few suspect components.

Yaoci Agent

The alibabacloud-yaochi-agent (Yaoci Agent) provides deep‑dive diagnostics for Alibaba Cloud databases such as RDS, PolarDB, Tair (Redis‑compatible), MongoDB, Lindorm, AnalyticDB, ClickHouse, and SelectDB by querying real‑time metrics and logs.

Scenario 1 – L3 Diagnosis (RDS)

Alert: active sessions on RDS instance rm‑j6c3l32c730ti37ur surged for 17 minutes. The agent identified that many queries targeted the inventory table’s product_id column with correlated sub‑queries. Because the product_id index was missing, each query performed a full table scan, causing session buildup.

Root cause: missing index on inventory.product_id . Remedy: add the index to eliminate full scans.

Diagnosis time: a few minutes (L3).

Scenario 2 – L4 Diagnosis (Redis Lua Script)

Alert: frontend latency spikes while Redis metrics showed CPU at 98 % and QPS dropping from 1200 to 180. The Yaoci Agent’s slow‑command log pinpointed a Lua script that repeatedly executed heavy calculations, monopolizing the single Redis thread.

Root cause: a Lua script blocked the Redis thread. Remedy: refactor the script to remove heavy computation or split work with pipelines.

Diagnosis time: 5–10 minutes (L4).

Scenario 3 – L4 Diagnosis (Redis EVAL)

Alert: similar frontend slowdown, Redis CPU saturated, QPS fell, and the slow‑command log revealed repeated EVAL commands containing large inline logic. These commands occupied the Redis thread, queuing all other operations.

Root cause: EVAL commands dominated the Redis thread. Remedy: locate the offending client, block the calls, and move complex logic out of Redis.

Both Redis scenarios share the symptom “slow command monopolizes the single thread,” but the remediation differs, illustrating command‑level diagnosis precision.

Comparison

Scenario 2 stems from a persistent business Lua script; Scenario 3 is caused by ad‑hoc EVAL calls. The agent’s ability to pinpoint the exact command and its origin enables targeted fixes rather than generic performance tuning.

Supported Databases

The Yaoci Agent currently supports Alibaba Cloud databases: RDS, PolarDB, Tair (Redis‑compatible), MongoDB, Lindorm, AnalyticDB, ClickHouse, and SelectDB. It does not support self‑hosted or non‑Alibaba databases.

Quick Start

Log in to the STAROps console at https://starops.console.aliyun.com/ and create a digital employee.

In the Skill Center, search for alibabacloud-yaochi-agent and attach it to the employee.

Ask a question, e.g., “Analyze why RDS instance rm‑j6c3l32c730ti37ur had a spike in active sessions between 2026‑07‑18 12:21 and 12:38.”

Single‑instance diagnostics return in minutes; cross‑layer root‑cause analysis typically takes 5–10 minutes.