Automating Fault Localization at 10M QPS: Evidence Chains Over Manual Hunts

Why Manual Troubleshooting Fails at Scale

Manual localization involves four actions: confirm impact, narrow scope, form hypotheses, verify root cause. It's a mature reasoning process but relies on human short-term memory and local knowledge. At 10M QPS, the search space explodes: 300 services × 40 instances × 80 metrics = millions of observable objects. A request traverses 10–20 components; any upstream anomaly cascades into downstream alerts. Each diagnostic step offers ~5 candidate directions; six steps yield 15,625 theoretical combinations. Engineers prune by experience, but novel faults exhaust time.

A localized fault leaves multi-layer symptoms. Monitoring shows multiple red dots; the localization system must reconstruct temporal and dependency order among them. Sorting by alert severity alone surfaces gateway and payment services first, burying the true early change (e.g., a config push).

Organizational Boundaries Break Evidence Chains

Modern systems span gateway, business, middleware, and infrastructure teams. Each sees a true but partial view. Chat-based evidence compresses nuance: "database is fine" may mean only CPU/connections look normal; "no deploy" may miss config, feature flags, certs, or scheduler changes. Such vague statements prematurely exclude valid directions.

High Pressure Amplifies Cognitive Bias

Engineers anchor on the first strong signal (recent deploy, high-CPU host, striking log). New evidence is interpreted to support the initial hypothesis; counterexamples are ignored. Familiarity bias drives investigators toward components they know best, but faults ignore team boundaries.

Automation doesn't replace human reasoning—it handles the mechanical, expensive search, correlation, and sorting so engineers can focus on verification and decision-making.

What Automated Localization Must Actually Answer

"Find the root cause" conceals at least five distinct questions. If the system only says "order service abnormal," it's anomaly detection. If it says "order service is the common upstream of multiple symptoms," it's candidate ranking. If it adds "config version c918 covered only the faulty instances and rollback restored metrics," it approaches root-cause verification.

Automated localization should output evidence chains and confidence scores, not an unexplained component name. In production, many changes co-occur without relation: CPU rise may be a retry side effect; slow queries may be secondary to traffic shift; a deploy may be coincidental. A usable system must express uncertainty and specify missing evidence.

From Fault Objects to Fault Hypotheses

Traditional dashboards use metrics or components as atomic units; automated localization should use hypotheses. A complete hypothesis includes trigger event, affected objects, failure mechanism, propagation path, and business impact. Example:

Config version c918 pushed at 02:16:32 to order-service v42's 12 instances, causing them to send inventory requests to a decommissioned address, triggering connection timeouts and payment retries.

This structure lets the system seek evidence for each clause and flag which links remain inferred—far more verifiable than "order service may be abnormal."

Build the Evidence Foundation Before Algorithms

Many projects start with graph algorithms or LLMs but stall on data mismatches. Metrics call a service order-api, traces call it order-service, the release platform uses repo name orders, logs only record pod names. Without unified identity, algorithms analyze fragments.

Unified Entity Identity

The platform must identify at minimum: services, instances, deployments, config items, feature flags, certificates, nodes, zones, network paths, storage volumes, and data partitions. Instance identity is critical: IP alone conflates container lifecycles. Safer to retain workload UID, instance UID, start time, and version so the system knows two processes on the same IP are distinct entities.

Temporal Consistency

Localization depends on precedence, which depends on time. Collector clock skew of tens of seconds can invert cause and effect. Beyond strict NTP, record event time (when it happened), collection time (when agent observed it), and ingestion time (queue + processing latency). Localization windows should use event time and model source latency; otherwise a change record delayed 40 seconds appears after the alert.

Bring Changes Into the Primary Evidence Chain

A large share of incidents relate to changes, but "change" extends beyond code deploys: config, flags, traffic routing, permissions, certs, dependency versions, data fixes, node migrations, autoscaling. Every change must record: initiator and timestamp, target objects and scope, before/after values or versions, canary batches and rollback linkage, automation identity, and linked tickets/approvals/pipelines. Without scope, a change record is low value—knowing "order service deployed" doesn't tell if anomalies concentrate in new-version instances. Measuring overlap between anomaly set and change set dramatically strengthens evidence.

Preserve Correlation Context in Telemetry

Metrics show trends, logs explain discrete events, traces reconstruct request paths, continuous profiling reveals code hotspots—they complement, not replace. Use trace IDs to attach logs to requests, exemplars to jump from high-latency metrics to representative traces, and service/instance/version tags to link to change records. Tag cardinality must be managed: distinguish stable aggregation labels from on-demand detail fields.

The data foundation's core isn't "collect more" but ensuring every piece of evidence can answer: who it belongs to, when it happened, and what it affected.

How the Localization Engine Converges Symptoms into Candidates

With the evidence foundation, the engine works in four layers, each reducing candidates without forcing a single conclusion.

Step 1: Normalize Disparate Signals into Structured Anomaly Events

Raw metrics can't feed reasoning directly. A service may have hundreds of time series; one fault triggers many similar alerts. Normalize into structured events, e.g.:

Object: order-service / v42 / East China
Window: 02:16:40 to 02:23:10
Signal: inventory call timeout rate
Deviation: 8.6× relative to same-week baseline
Scope: 12 of 48 instances
Direction: increase
Confidence: 0.97

Baselines must combine static thresholds, seasonal baselines, peer-group comparison, and business rules. Peer-group comparison is especially powerful: if 12 new-version instances are abnormal while 36 old-version instances are normal, the differential is strong evidence even if global average barely moves. Conversely, if multiple unrelated services in one zone jitter together, candidates should shift to shared infrastructure.

Step 2: Use Temporal Relations to Eliminate Impossible Causes

Cause cannot follow effect—a cheap pruning rule. Record start-time distributions, not single points, because sampling intervals, smoothing windows, and reporting latency introduce error. Judge whether a candidate *probably* preceded the symptom.

Example: order instances turn abnormal 02:16:40–02:16:55; payment retries appear 02:17:12–02:17:25. Temporal order supports the former causing the latter. If a database's CPU rises only at 02:19, it's likely a secondary effect of retry traffic. Recovery order also helps: if config rollback restores order timeouts first, then payment retries, then gateway latency, the reverse propagation chain reinforces the hypothesis.

Step 3: Directed Pruning Along Topology

Topology tells "who can affect whom." It includes request dependencies, resource hosting, control relationships, and data dependencies. A practical fault graph needs at least four edge types:

Request dependency edges (e.g., gateway → order)

Resource hosting edges (e.g., instance runs on node)

Control relationship edges (e.g., config item pushed to instance group)

Data dependency edges (e.g., job reads table/partition)

From damaged business metrics, search backward for common upstreams. If payment, inventory, and coupon APIs all fail and share a gateway cluster, that common upstream warrants priority. If only the inventory API fails, scope should shrink to interface-level dependencies—don't flag the entire order service.

Topology must be time-versioned. Deploys, scaling, and service discovery constantly mutate call graphs. Using current topology to explain past faults introduces edges that didn't exist then. Practical approach: retain periodic snapshots and edge validity periods to reconstruct the fault-window graph.

Step 4: Multi-Evidence Scoring

Candidate ranking can start with an explainable weighted model. Illustrative weights (not for direct production use):

Weights must be calibrated per business signal reliability using historical incident labels. The system must show which evidence contributed to a score so engineers understand why a candidate ranks first.

Correlation ≠ Root Cause: Adding Causal Constraints

The gravest failure isn't "not found" but confidently wrong. At scale, many things happen simultaneously; pure correlation, temporal proximity, or log similarity easily produce plausible pseudo-root-causes.

Explain "Clustered Anomalies" with Common Cause

If 20 services in one zone simultaneously spike latency, they didn't all fail independently. They may share a network path, DNS resolver, node pool, or storage system. The engine should seek the candidate that explains the most symptoms while introducing the fewest extra assumptions—akin to minimal cut sets in fault trees. A candidate at the intersection of multiple anomaly paths gets priority, but intersection is necessary, not sufficient; the candidate itself must show changes consistent with the failure mechanism.

Approximate Counterfactuals via Scope Comparison

Causal question: "Would the fault still occur without this change?" Production experiments are hard, but near-counterfactuals exist:

Old-version instances of the same service

Canary groups that didn't receive the config

Unaffected availability zones

Historical periods with similar traffic profile

Degraded requests that bypass a dependency during the incident

If new-version instances show 4.2% error rate vs. 0.03% on old versions with comparable traffic, the version change explains more than "global CPU rise." If rolling back a few instances instantly restores them while non-rolled-back instances stay broken, evidence strengthens further.

Write Intervention Results Back into the Evidence Graph

Isolation, degradation, traffic shifting, and rollback aren't just remediation—they're causal experiments. The platform should record: affected objects, start time, expected vs. actual outcome. This closes the "hypothesis → intervention → observation → conclusion" loop.

Interventions are risk-constrained. Can't restart a database at peak to verify root cause. The system can recommend verification actions, but execution requires approval, canary, timeout, and auto-rollback. For high-risk components, observational evidence beats active experiments.

Automated localization truly participates in reasoning only when it actively lowers confidence upon seeing counter-evidence.

From Rules to Graph Models to LLM Collaboration

No single technique is the endpoint. Rules, statistical models, graph algorithms, and LLMs each solve different problems; a practical system layers them.

Rules Encode Deterministic Knowledge

Rules excel at stable, explainable engineering knowledge: "change after fault → not trigger," "single instance abnormal + multiple services on same node abnormal → check node first," "errors only on specific version → boost version candidate." Rules are controllable, auditable, fast to cold-start. Downside: limited coverage; rule conflicts grow with count. Use rules for hard constraints and high-value patterns—not to enumerate all faults.

Statistical and Graph Models Discover Structure

Statistical models handle seasonal baselines, change-point detection, peer-group differences, lead-lag analysis. Graph algorithms find common upstreams, match propagation paths, compute centrality, aggregate fault domains. Combined, they yield structured candidates without natural language descriptions.

With enough quality-labeled historical incidents, a ranking model can learn per-business evidence weights. Labels must go beyond a single "root cause" line in postmortems; they should include trigger event, fault object, propagation path, evidence, interventions, and final conclusion.

LLMs Organize Evidence, Not Adjudicate

LLMs add value in three stages: (1) map natural language from log templates, change descriptions, tickets, postmortems to unified entities and fault patterns; (2) generate readable hypothesis explanations from structured evidence, reducing cross-platform context switching; (3) retrieve similar past incidents and suggest next verification questions.

LLMs must not bypass the data layer to read thousands of raw logs and declare root cause. Context is incomplete; logs contain noise and misdirection. Let deterministic systems handle entity resolution, time alignment, topology search, and numeric computation first, then feed compressed evidence to the LLM.

LLM outputs need citations. Every conclusion must trace to a specific metric window, log event, trace sample, or change record. Missing citations → mark as "pending verification inference," not established fact.

Architectural Shifts at 10M QPS

From 100K to 1M QPS, automation solves efficiency: unify dashboards, logs, releases to reduce manual hops. At 10M QPS, it must also control the observation and computation storm generated by the fault itself.

The Localization Platform Must Withstand Fault Traffic

During incidents, error logs, retry traces, and alert counts surge. If the localization platform shares fragile dependencies with the business, it becomes unavailable when needed most. Collection pipelines need rate limiting, sampling, backpressure, and degradation; query layer needs pre-aggregation and caching; critical SLIs and change streams get higher priority.

Trace sampling must account for fault bias. Fixed-rate sampling may miss low-frequency high-value errors; increasing sampling after errors appear can overwhelm the backend. Combine head-based baseline sampling, tail-based error retention, per-tenant budgets, and dynamic caps to balance information value against system load.

Descend from Service-Level to Instance and Interface Topology

At 1M QPS, whole-service anomalies draw attention. At 10M QPS, faults are often partial: specific version, batch of instances, shard, zone, user segment, or API endpoint. Global averages dilute the signal; service-level topology hides true boundaries.

The localization system needs multi-granularity graphs. Normally aggregate at service level; on anomaly, drill down by version, instance, interface, datacenter, or shard. Keeping all dimensions at high precision constantly is cost-prohibitive; use a "coarse filter then drill down" tiered strategy.

Protect Evidence Quality Amid Telemetry Storms

A major incident can generate billions of logs or spans in minutes. Blind dropping breaks causal chains; keeping all is infeasible. Tier by evidence value:

Cross-Domain Faults Need Federated View

Large systems scatter observability data across regions, clouds, and compliance domains. Central aggregation is expensive and fragile. Let each domain locally normalize anomalies and generate preliminary candidates, sending only compressed events, topology summaries, and evidence references to the global layer.

The global layer hunts cross-domain common causes (shared config control plane, shared certificate, shared traffic policy). Drill-down to domain detail happens on demand with proper permissions. This reduces transfer pressure and avoids making the central platform a single point of failure.

The essential difference at 10M QPS isn't just more candidates—it's that the fault simultaneously mutates business traffic, telemetry traffic, and the localization system's own workload.

Phased Rollout: Avoid Building a "Universal Root-Cause Platform"

Spending a year building a unified platform before validating on real faults rarely succeeds. Start with high-frequency, evidence-rich fault types and expand automation incrementally.

Phase 1: Structure the Manual Evidence Path

Don't chase automatic answers yet. Integrate the on-call engineer's common entry points so each event auto-attaches:

Affected business SLIs, regions, interfaces, versions

Code, config, and traffic changes in the ±30 min window

Upstream/downstream topology of anomalous services

Peer comparisons: same version, same zone, same node

Representative error traces and log templates

Success metrics: fewer page jumps, shorter time to first hypothesis, higher change-record coverage. Even without complex algorithms, this sharply cuts triage cost.

Phase 2: Candidate Ranking with Evidence Explanation

Pick a few well-bounded fault classes: deploy regressions, config errors, single-node anomalies, single-shard hotspots. Use time, scope, topology, and peer comparison to generate candidate lists with supporting and contradicting evidence per candidate.

Don't only measure "is rank-1 equal to postmortem root cause." Track top-3 recall for frequent faults, misleading high-confidence output rate, evidence completeness, and why engineers accept or reject candidates.

Phase 3: Integrate Low-Risk Verification Actions

Once ranking stabilizes, add read-only queries and controlled verifications: instance comparison, targeted trace capture, increased sampling on a slice, shifting 1% traffic to a healthy pool. Actions must have budgets, permissions, timeouts, and auto-reclamation.

For stateful actions (config rollback, instance isolation), the system proposes the action and expected outcome; the on-call confirms execution. Observed results auto-write back to the event, updating candidate confidence.

Phase 4: Continuous Learning Loop

Postmortems shouldn't just produce a document. After resolution, guide the owner to annotate:

Final trigger event and failure mechanism

Which symptoms were propagation effects

Which candidates were red herrings

Which verification action was most discriminative

Which telemetry or topology was missing at the critical moment

These annotations tune rules, calibrate scores, and fill data gaps—not merely to train a "root-cause classifier." Next time a similar fault occurs, the system should take one fewer detour.

Constrain Automation Quality with Metrics

Evaluate the localization system on:

In a pilot, a sample target: 80% of high-frequency faults yield a top-3 candidate containing the true root cause within 5 minutes, with all high-confidence conclusions backed by at least two independent evidence types. These numbers are project-specific examples, not industry benchmarks. Calibrate gradually based on fault distribution, data quality, and remediation risk.

The Endpoint of Automation: Faster Falsifiable Judgments

Fault localization doesn't jump from manual to fully automatic via one algorithm. It's an evolution path: unify identity and time first, then integrate changes and topology; first auto-collect evidence, then rank candidates; first assist verification, then consider controlled remediation.

At smaller scale, a business-savvy engineer with clear dashboards and runbooks is often enough. As scale grows, automation's value emerges because it stably handles the massive search spaces and cross-domain correlations humans struggle with. But regardless of scale, final remediation must weigh business impact, data risk, and recovery cost.

A good localization system doesn't save you from thinking—it accelerates the problem into a few falsifiable judgments.

When the next alert storm hits, do you want the system to just say "37 components abnormal," or to hand you an evidence chain and explicitly state which step still needs verification?