From Blame Games to Causal Chains: How AI Agents Revolutionize Ops Troubleshooting

Modern distributed systems often produce alerts in one technical domain while the root cause resides in another. Traditional troubleshooting relies on multiple teams each proving their own innocence — application, database, middleware, network, and storage teams check their own dashboards and find nothing wrong, yet the failure persists. This article shows how Alibaba Cloud's STAROps (a cross-domain intelligent operations platform) and OSS Agent (a storage-specialized agent) work together to stitch scattered signals into a complete causal chain, moving ops from reactive blame-shifting to precise diagnosis.

Scenario 1: Order Interface Timeout at Promotion Midnight — 503 SlowDown with Normal Bucket Metrics

At 00:00 during a major promotion, the order service error rate hit 23.6% and P99 latency 8.2s. Uploads of payment vouchers to OSS returned 503 SlowDown. Database, cache, message queue, and network all checked out normal. The storage team confirmed bucket-level request count, bandwidth, and QPS were far below limits — no throttling triggered at the bucket level.

Cross-Domain Correlation by STAROps

STAROps placed monitoring from all five domains on a single timeline. Only the application-side error rate and OSS 503 responses rose in lockstep; the other four domains showed zero fluctuation. The fault was isolated to OSS.

Deep Storage Diagnosis by OSS Agent

The agent examined the 503 error semantics: SlowDown indicates throttling, but bucket-level QPS, bandwidth, and request count never hit thresholds. It then compared the application write rate (daily ~800 QPS, spiking to 6,200 QPS at midnight) with request distribution and drilled down one layer: which layer under the bucket is being throttled?

Investigation reached the partition layer:

OSS auto-partitions by UTF-8 order of object names. Throttling can hit a single partition while bucket-level aggregates remain healthy.

All 503 errors concentrated under prefix voucher/20260810/, accounting for 97% of bucket write requests.

Naming rule was voucher/{yyyyMMdd}/{auto-increment}.jpg — a classic sequential prefix . All incremental writes landed on lexicographically adjacent partitions.

The hot partition exhausted I/O; OSS throttled that partition and returned 503 (error code 0026-00000028). Bucket-level metrics use the whole bucket as denominator, so post-aggregation they appeared far below limits.

Root cause: Sequential prefix naming concentrated promotion writes onto a few partitions, triggering partition-level throttling invisible to bucket-level monitoring.

Remediation: Emergency — exponential backoff retry for 503; quick fix — add a random factor to object names (one-line change) to scatter writes; fundamental — switch to hashed prefixes, e.g., voucher/{hash(uid)%256}/{yyyyMMdd}/{uid}.jpg. Three other buckets using the same naming convention were fixed together.

Scenario 2: Avatar Upload Success Rate Drops to 61% — 403 AccessDenied with Valid STS/RAM

After a Friday night security hardening that added a bucket policy restricting access to a specific VPC (approved and tested in staging), Monday morning avatar upload success fell from 99.9% to 61%. Clients used STS temporary credentials for direct upload. App team confirmed STS credentials valid, RAM permissions unchanged, no recent deployment — standard permission checks couldn't explain the selective failures.

Cross-Domain Correlation by STAROps

STAROps correlated the 403 source IP distribution: all failed requests came from public IP ranges; VPC-internal calls had zero failures. The failure surface aligned with network origin, not user or interface — ruling out credential expiry or code bugs. It then pulled audit logs and found a PutBucketPolicy operation at Friday 22:10 by a security admin. The change sat dormant over a low-traffic weekend, triggering no alerts.

Deep Storage Diagnosis by OSS Agent

The agent evaluated OSS's authorization decision order layer by layer:

ACL: public-read — allow, excluded.

Referer anti-leech: whitelist empty and empty Referer allowed — full allow, excluded.

Bucket Policy: matched a Deny statement — root cause here .

The offending Deny used StringNotEquals on acs:SourceVpc, rejecting requests not originating from the specified VPC. This matched the observed public-IP failure distribution. In OSS's evaluation order, an explicit Deny takes precedence over Allow in RAM policies, so valid STS credentials and correct RAM permissions still resulted in 403.

Based on 30-day access logs, the agent identified two additional public callers blocked by the same rule: a cross-region data sync job and a partner callback service — neither had reported issues yet.

Root cause: New bucket policy explicitly denies non-VPC sources; clients use STS from public internet. Explicit Deny overrides RAM Allow.

Remediation: Emergency — add exceptions for affected roles/callers in the Deny rule or roll back; fundamental — inventory all callers and source networks from 30-day access logs, redesign policy, roll out in phases, and embed "impact assessment from real access data" into the change process.

Scenario 3: Anti-Leech Enabled but Traffic Surges 17x — Bandwidth Saturation and Cost Spike

An enterprise served app installation packages via OSS. Monday morning finance reported outbound traffic package exhausted (switching to pay-as-you-go), and customer service reported slow downloads. Monitoring showed traffic spiked to 17x baseline starting Saturday midnight. No business traffic growth, no weekend deployment, anti-leech was enabled.

Cross-Domain Correlation by STAROps

STAROps aligned three signals — finance alert, customer complaints, monitoring curve — to the same timestamp and bucket. Drilling by object revealed anomalies concentrated on a single object.

Deep Storage Diagnosis by OSS Agent

The agent inspected the anti-leech configuration: the switch was indeed ON. It then examined the switch's actual decision behavior:

Object app-release.apk contributed 89% of outbound traffic.

94% of requests for this object carried no Referer, originating from 3 IP segments with User-Agent indicating a downloader tool.

Anti-leech was enabled, but "Allow empty Referer" was ON — these requests passed directly. Common sources of empty Referer: browser address bar, wget / curl, HTTPS page referencing HTTP resource (Referer stripped), App WebView privacy policies.

Download speed dropped from 8 MB/s to 600 KB/s, matching the traffic surge. Abnormal traffic saturated bucket egress bandwidth, triggering OSS flow control; legitimate downloads were squeezed into remaining bandwidth.

The agent proposed three mitigation strategies, ranked by business intrusion:

Aggressive: Disable "Allow empty Referer". Not recommended for emergency — app version-check API itself sends no Referer; disabling would break that feature immediately.

Moderate: Use bucket policy to temporarily deny the 3 offending IP segments. Recommended for this case — instant stop without affecting normal access. However, attackers can switch IP segments; this is stopgap, not cure.

Conservative: Configure traffic surge alerts while rolling out signed URLs / CDN URL authentication. No false positives , but already-incurred costs and degradation won't revert instantly; root fix requires scheduling.

Recommended approach: moderate for immediate stopgap, conservative for permanent fix.

Root cause: "Allow empty Referer" exception let hotlink requests without Referer pass through, generating extra cost and consuming bandwidth that slowed legitimate downloads. A switch opened for the company's own app became an open backdoor for all downloaders.

Common Pattern Across Three Scenarios

Symptoms differed, but two commonalities emerged:

The alert domain and the root-cause domain are misaligned. In each case, the team owning the alerting domain saw nothing wrong.

Every root cause was a configuration that looked reasonable when applied. Sequential prefixes for daily partitioning, a security hardening policy, an anti-leech exception for app compatibility — all made sense at the time.

Such failures cannot be solved by adding more people or more dashboards. They require stitching data scattered across technical domains, teams, and time points into a single causal chain. STAROps handles cross-domain correlation; OSS Agent provides storage-layer expertise. Together they prevent the investigation from stalling at unhelpful conclusions like "OSS returned 503" or "OSS returned 403" — conclusions that leave the on-call engineer at 3 AM no closer to a fix.