Six Dimensions to Classify Faults at 10M QPS: From Guesswork to Structured Facts

Introduction: The Problem of Ambiguous Fault Labels

At 02:17 AM, alerts flood the on-call channel: entry success rate drops, a data center's connection count fluctuates, database slow queries rise, and message queue consumer lag increases. The on-call engineer asks: "Is this a network fault, a database fault, or a traffic fault?" Multiple hypotheses emerge—some blame the database, others cite high single-host load suggesting capacity shortage, another proposes traffic shifting first. Each has supporting evidence yet explains only part of the phenomenon. Fifteen minutes later the team discovers the true origin: a configuration change routed a small fraction of requests onto an abnormal retry path, which then amplified connection-pool contention and queue backlogs.

Such scenes are common. Engineers aren't lacking experience; each simply uses a different classification mental model—by component, by symptom, by cause, or by remediation action. Mixing these dimensions into a single label like "database fault" makes alignment impossible. At small scale a senior engineer can mentally reconstruct context. At ten million QPS across regions and clusters, anomalies appear in combination and propagate faster. Relying on a vague label to launch response can misdirect remediation from the first minute.

Fault classification is not about sticking a pretty label on an incident; it is about converting an uncertain scene into structured facts that are computable, routable, and actionable.

Why Experience-Based Classification No Longer Scales

Experience Relies on Implicit Context

A statement like "looks like a cache avalanche" hides a long chain of unspoken observations:

Hit rate dropped from 96% to 61% in five minutes

Origin database read traffic grew 4.8×

Large numbers of cache keys shared the same expiration window

Application CPU did not drop before hit rate fell

No recent cache client release

If only "cache fault" is recorded, the next on-call engineer cannot verify the reasoning nor turn the experience into a detection rule. The classification result appears concise but discards the most valuable evidence.

One Word Can Mean Four Different Things

Teams often conflate:

Component (where the metric looks bad)

Symptom (what users or systems observe)

Cause (why it happened)

Remediation (what we plan to do)

"Database fault" might mean connections exhausted, or application retries dragging the database down, or merely that the user-visible symptom landed on database metrics. A single label carrying cause, phenomenon, and impact prevents stable information flow.

Scale Amplifies Combinations, Not Label Count

A monolith has limited components and short propagation paths. In a distributed system a request may traverse gateway, service discovery, multiple services, caches, databases, and message systems—each deployed across data centers, clusters, and versions. With 200 service types, 5 regions, 3 release versions, and 8 common failure modes, potential combinations reach 24,000. Real systems add tenants, traffic types, dependency directions, and durations. Hand-writing a unique fault name for every combination is impossible. The pivot from experience to system is to establish a set of orthogonal dimensions that let any scene be described by combination.

Define the Boundaries: What Exactly Is a Fault?

From Defect to Impact Is a Chain

Engineering contexts often confuse fault, error, and failure. The article models fault evolution in three layers:

Fault Factor : Internal or external condition that may lead to anomaly (misconfiguration, disk corruption, dependency overload).

Error State : The factor enters runtime and deviates internal state from expectation (missing routing table entry, connection pool exhaustion, primary-replica replication fork).

Service Failure : The error state crosses the system boundary and breaks an external promise (requests exceed SLO, wrong results returned, data unavailable).

A fault factor does not necessarily become a service failure—misconfiguration may be caught by a validator, single-disk loss absorbed by replication. Conversely, one service failure may stem from multiple factors. Classification should prioritize observed error states and service failures; root cause remains a continuously updated hypothesis, not an answer required in the first minute. Forcing an immediate root-cause choice anchors the team to an early guess and resists later evidence.

System Boundaries Decide "Internal Fault" vs "External Dependency"

The same anomaly falls into different categories for different teams. The payments team sees the bank channel as external; the platform team sees the payments service as downstream; the business team only cares whether the payment capability meets its promise. Every classification record must therefore make three boundaries explicit:

Service boundary : Which external capability is being evaluated

Ownership boundary : Which team currently holds remediation authority

Observability boundary : Whether evidence comes from inside the service, the dependency side, or the user side

With boundaries clear, "external dependency fault" becomes a routable fact: the current service cannot fix the dependency directly, so it must execute isolation, degradation, failover, or coordinated upgrade.

Fault Is Not a Binary State

Systems often operate in a gray zone—"still serving but already deviating from promise." Examples:

99.9% requests healthy, but tail latency in one region persistently exceeds target

Main path available, yet fallback path data freshness degrades

Reads normal, writes intermittently time out

Large-tenant customers affected while regular tenants appear fine

If classification only has "healthy" and "faulty," these partial degradations are masked by averages. A better approach evaluates availability, correctness, timeliness, completeness, and security separately, attaching scope and severity to each classification.

Six Orthogonal Dimensions for a Fault Coordinate System

An extensible taxonomy should not force a single pick from a long list. A practical model maps each event to multiple, mutually independent dimensions.

Dimension 1: At Which Layer Did the Fault Occur?

"Occurrence layer" narrows ownership but does not equal root cause. Common layers include:

Access / Gateway layer

Control plane (config, service discovery, scheduling)

Application / Business logic layer

Data layer (cache, database, message queue)

Infrastructure layer (network, storage, compute)

The value of layer classification is routing: if evidence points to control plane, engage config or release platform owners immediately instead of letting the database team chase slow queries for half an hour.

Dimension 2: How Did the Capability Fail? (Failure Mode)

Failure mode describes "what it looks like when broken" and is closer to user perception and remediation action than component name:

Unavailable : Requests cannot complete or capability fully interrupted

Performance Degradation : Latency, throughput, or resource efficiency deviates from target

Incorrect Result : Request completes but returns wrong content

Data Inconsistency : Conflicting results across replicas, regions, or read paths

Stale Data : Result correct but fails freshness requirement

Functional Degradation : Core function works, subset of capabilities disabled or substituted

Security Failure : Authorization, isolation, confidentiality, or integrity promise broken

This dimension deliberately avoids "why." Connection pool exhaustion, packet loss, and thread blockage can all manifest as performance degradation or unavailability. Identifying failure mode first lets protective actions run before root cause is confirmed.

Dimension 3: How Large Is the Impact Scope?

Remediation priority correlates directly with blast radius. Scope can be recorded across multiple facets:

Region: single AZ, single data center, single region, cross-region

Service: single instance, single shard, single cluster, global

Traffic: single API, single business line, single tenant, specific client version

Proportion: affected request ratio, user ratio, data ratio

Value: whether core transactions, funds, identity, or security capabilities are involved

"Affects 1% of requests" is not always low priority. If that 1% are all payment confirmations, business consequence may exceed a 20% latency rise on a generic query endpoint. Scope classification must be used together with business impact.

Dimension 4: How Does the Anomaly Evolve Over Time? (Temporal Pattern)

Temporal shape is critical for diagnosis and response:

Transient Spike : Shorter than observation window; likely jitter or burst traffic

Sustained Anomaly : Persists across multiple detection cycles; stable trend

Periodic Anomaly : Correlated with cron jobs, traffic cycles, or resource reclamation

Gradual Degradation : Metrics slowly worsen; typical of leaks, backlogs, capacity approach

Intermittent Anomaly : Alternating occurrence and recovery; causes alert flapping

Permanent Failure : Will not self-recover without external action

Same latency rise: transient spike suits observation and rate-limiting; gradual degradation demands capacity and leak assessment; periodic anomaly should first align with scheduler and batch timelines.

Dimension 5: Can It Be Directly Observed? (Observability / Confidence)

Observability determines classification confidence and remediation risk. Allowing "unknown" is vital—it is not a classification failure but an honest statement of evidence quality. The system can raise sampling, expand diagnostic data, or escalate to human analysis instead of forcing the event into a familiar bucket.

Dimension 6: What Business Promise Is Impacted?

Technical metrics must map to service commitments. At minimum record:

Is availability below SLO?

Has latency crossed target percentile?

Is result correctness suspect?

Are data completeness and freshness compromised?

Do RTO/RPO risks exist?

Are compliance, security, or financial risks triggered?

After these dimensions, an event is no longer just "cache fault" but can be described as:

Data layer, performance degradation + stale data, single region 12% read requests affected, gradual, correlated inference, medium business impact, current confidence 0.78.

This description looks longer than a single label, yet it directly drives routing, priority, protective actions, and subsequent analysis.

Separate Symptoms, Causes, and Evidence in the Record

Many classification systems degenerate into another alert naming scheme because they jump from a single metric to a conclusion. Example: "CPU > 90%" classified as "compute resource shortage." This step seems reasonable but omits critical reasoning. High CPU could be traffic growth, infinite loop, compression task, GC, retry storm, or the monitoring agent itself. Resource shortage is only one explanation. If automation treats it as confirmed root cause and immediately scales out, it may temporarily mask the anomaly or amplify erroneous traffic to more instances.

Event Model Contains at Least Five Field Groups

An auditable classification event can be divided into:

Fact fields : timestamp, object, region, version, changes, metric values

Symptom fields : user-visible and system-visible anomalies

Classification fields : occurrence layer, failure mode, scope, temporal pattern, business impact

Hypothesis fields : candidate root causes, supporting evidence, contradicting evidence, confidence

Action fields : executed mitigations, expected effect, actual result, rollback conditions

Facts do not change with reasoning; hypotheses are allowed continuous revision. Separating them lets the team preserve the full cognition process during incident evolution.

Evidence Must Carry Source and Freshness

Not all evidence is equally reliable. User-side real requests, independent synthetic probes, service-internal metrics, and human descriptions have different trust boundaries. Old evidence may become invalid after system switches. Attach these attributes to evidence:

Source: user-side, service-side, control-plane, dependency-side, human observation

Time window: start and end of evidence coverage

Freshness: age relative to now

Completeness: fraction of regions, instances, or traffic covered

Consistency: whether corroborated by other independent signals

Sampling bias: possibility of missing long-tail or specific tenants

The classification engine need not start with a complex model. Even explicitly recording "two independent signals agree and cover 80% of affected traffic" is more reliable than a single threshold trigger jumping straight to a conclusion.

Counter-Evidence Is as Important as Supporting Evidence

Confirmation bias is rampant in war rooms: once someone says "database slow," everyone hunts for signals supporting database fault. Systematic classification must enforce retention of counter-evidence, e.g.:

Database latency up, but only on connections from a specific application version

Downstream errors rise, yet synthetic probes directly hitting downstream remain healthy

Single-host CPU high, but traffic migration did not restore latency

Counter-evidence doesn't instantly give the answer, but it eliminates wrong hypotheses. A taxonomy that only collects supporting evidence merely codifies experience bias into the system.

How Classification Drives Detection and Remediation

The measure of a taxonomy is not how complete its fields are, but whether it makes the next step faster and more stable.

Start with Signal Normalization

Different systems emit signals in varied formats. Metric alerts, log anomalies, traces, change events, user feedback, and synthetic probes must first be normalized into a unified event. Normalization does not discard raw data; it extracts comparable core fields. After normalization the system can recognize that seemingly different alerts—gateway timeout, service retry, database connection rise—in the same region, sharing topology and request tags, are highly correlated in time and should be aggregated into one event instead of paging three separate teams.

Classification Is a State Machine, Not a One-Shot Decision

A fault's classification evolves with evidence:

Initial: only user-side timeouts → "access or dependency unknown, partial unavailable" Trace data shows timeouts concentrated between Service A and Service B → update occurrence layer Change log reveals Service A just adjusted retry policy → add candidate root cause Rollback restores error rate → root cause confidence rises Postmortem finds missing config validation → add upstream management cause

Therefore classification records must be versioned, not overwritten. Every change retains timestamp, triggering evidence, and operator. This supports both real-time response and later rule training and model evaluation.

Confidence Determines Automation Boundaries

Automation isn't "automate everything possible." A more rational approach uses impact and confidence jointly to decide action:

High impact, low confidence → most dangerous zone. System must rapidly assemble cross-team responders while avoiding irreversible operations.

Low impact, high confidence → suitable for automation: isolate single anomalous instance, increase diagnostic sampling, pause batch of non-critical tasks.

The core of automation is not replacing human root-cause guessing, but stably executing pre-validated actions once clear classification and risk boundaries exist.

Classification and Playbooks Should Be Many-to-Many

One classification may map to multiple mitigations depending on scope, phase, and business constraints; one playbook may cover multiple fault sources. Example: "protect downstream" applies to retry storms, traffic surges, and dependency performance degradation. Don't turn the classification tree into a rigid decision tree. Use policy conditions instead:

Failure mode = performance degradation

Impact scope > single instance

Downstream saturation high

Current classification confidence > threshold

Protective action recently exercised in drill

Action has explicit rollback condition

When conditions satisfy, system recommends or executes the action. Taxonomy stays stable; remediation policies iterate independently.

Why Classification Undergoes a Phase Change at Ten Million QPS

Going from one million to ten million QPS is not merely alerts ×10. Topology, traffic, and organization change simultaneously, presenting fundamentally different classification challenges.

Local Anomalies Are More Easily Masked by Global Averages

Assume 600M requests/minute globally. One region, one tenant suffers 1.2M failed requests/minute. Global failure rate is only 0.2%, possibly still within overall SLO, but for that tenant it's total unavailability. Larger scale → smoother aggregated metrics → local problems more easily hidden by "average correctness." Classification must preserve slices by region, tenant, version, API, and call path; otherwise detection sees a healthy dashboard while users see sustained failure.

Retries Decouple Fault Source from Pressure Source

In small systems the busiest component is often the problem center. In large call chains the busiest component may simply be absorbing upstream retries. Example: original traffic 8M QPS, a dependency exhibits 2% timeout. If two upstream layers each retry up to 2×, worst-case extra requests amplify along the chain. Actual amplification depends on success distribution, timeout windows, and backoff strategies—cannot apply a fixed multiplier—but even 10–20% increase can push a near-watermark downstream into saturation. Database connections and CPU both rise; classifying as "resource overload" is not wrong but incomplete. The system must also mark it as secondary pressure with the original failure on another dependency path.

Control Plane Anomalies Can Span Vast Data Planes

Config center, service discovery, certificates, release platform, scheduler—request volume may be low, yet a single fault can hit tens of thousands of instances simultaneously. Classifying only by data-plane QPS and resource watermarks underestimates control plane blast radius. Taxonomy needs a distinct control-plane identifier, recording change batch, propagation scope, and version consistency. A config error pushed to 1% of instances vs. the same error fully propagated demand completely different remediation urgency.

Multi-Active Architecture Makes "Recovery" Non-Trivial

In cross-region systems, traffic shifting may quickly restore availability but introduces capacity, data consistency, and cost risks. Fault classification should simultaneously express:

Current failing regions vs. healthy regions

Data replication direction and lag

Safe traffic capacity of standby region

Whether failover breaks session or idempotency boundaries

Conditions required for failback

"Region fault, shift traffic" is an experience slogan. Systematic classification hands fault type, capacity constraints, data risks, and switch conditions together to the remediation policy.

Alert Storms Demand Clustering Before Classification

Large incidents can trigger thousands of alerts in minutes. Classifying each individually is too slow and creates noise. Better sequence: first cluster by time correlation, topology proximity, shared tags, and change association; then classify the aggregated events. The goal isn't perfect root cause but compressing "3,000 alerts" into "one main event, three possible propagation branches, and some independent noise." On-call sees an actionable event graph, not a scrolling alert wall.

Making the Taxonomy Stick in Practice

Many teams design beautiful classification tables only to revert to free text months later. Common causes: too many dimensions, unclear ownership, classification disconnected from remediation.

Step 1: Derive Minimum Dimensions from Incident Samples

Pull the last 3–6 months of incidents, severe alerts, and user complaints. Don't rush to apply industry templates. For each sample answer:

What was the earliest signal observed?

Which piece of information changed the remediation direction?

Which labels had zero practical use?

Which misclassification cost the most time?

Which action could have been executed earlier?

If a dimension cannot change routing, priority, diagnosis, or remediation, it may not belong in the core taxonomy yet. Establishing 5–7 truly actionable dimensions beats designing 60 fields upfront.

Step 2: Define Controlled Vocabulary—But Keep "Unknown"

Core fields use controlled vocabularies to avoid "timeout", "request timeout", "timeout" becoming three categories. Each enum value needs:

Clear definition and counter-examples

Applicability boundary

Minimum required evidence

Allowed parent-child relationships

Owner and version

Simultaneously, must retain "unknown", "other", and free-text remarks. Without these exits, on-call engineers pick wrong labels just to submit the event, degrading data quality. High "unknown" rate should trigger investment in detection or vocabulary refinement—not blame the recorder.

Step 3: Automate Collection, Keep Judgment Explainable

Region, instance, version, recent changes, dependency topology, SLO status—fact fields should be auto-populated. Humans confirm business impact, add scene context, and review high-risk classifications. Automated classification output must explain "why":

Classified as "single-region dependency performance degradation" because 92% of timeouts concentrated in Region A, service-internal CPU normal, three independent synthetic probes accessing Dependency B all exceed latency threshold, no local service changes in last 20 minutes.

This explanation need not expose complex algorithms, but must let on-call verify evidence. Unexplainable high-confidence conclusions should not trigger high-risk actions directly.

Step 4: Embed Classification in the Incident Workflow

Classification should not be a postmortem form filled after the fact. It appears throughout the incident lifecycle:

Initial classification serves mobilization and protection

Investigation classification serves localization

Postmortem classification serves governance

All three phases can share the same dimension set, but confidence and granularity differ.

Step 5: Establish Taxonomy Governance

Vocabulary and rules evolve with architecture; governance ownership must be explicit:

Platform team maintains generic event model and tooling

Domain teams own domain-specific failure modes and evidence rules

SRE / stability team reviews cross-domain consistency

Incident owner confirms final classification in postmortem

Automation policy owners responsible for drill and rollback validation

Classification version must be saved with the event. Vocabulary changes need not rewrite all history—otherwise trends lose comparability. Build old-to-new mappings and convert at analysis layer.

Validate the Taxonomy with Three Real Scenarios

Scenario 1: Cache Hit Rate Drop

Initial symptom: hit rate 95% → 70%, database read traffic surges. Experience jumps to "cache avalanche." Systematic classification continues asking:

Is drop concentrated in a key batch, a shard, or a region?

Are misses increasing, or are requests not reaching cache at all?

Are expiration times clustered?

Are cache instance resources abnormal?

Did application version or routing just change?

Is user impact latency rise, errors, or still within SLO?

If evidence shows only new-version instances have wrong cache key prefix, classification becomes "application data access path error causing cache bypass, secondary database pressure"—not cache infrastructure fault. Remediation shifts from scaling cache to rolling back application version.

Scenario 2: Database Connection Exhaustion

Connection pool wait rises, DB connections hit limit—clear fact, yet not sole root cause. Classification distinguishes:

Is new request volume growing?

Is per-request connection hold time increasing?

Are abnormal retries present?

Are slow queries cause or secondary effect?

Is impact limited to certain APIs or tenants?

If call chain shows downstream timeout triggering transaction retry, connection exhaustion is a propagation node. Initial protection: limit retries, isolate anomalous traffic; then address downstream. Simply classifying as "database capacity fault" and scaling out may let retries amplify further.

Scenario 3: Message Backlog

Queue backlog can stem from producer burst, consumer slowdown, partition skew, broker anomaly, or downstream write slowdown. Systematic taxonomy records "backlog" as symptom, then uses throughput delta, temporal pattern, partition scope, and dependency state as evidence.

If backlog only on one partition and consumer CPU low → "partition-level consumption blocked" If all partitions grow synchronously while producer traffic unchanged → check consumer capacity or downstream dependency

Both scenarios may show identical total backlog volume yet require completely different scheduling and scaling strategies. The taxonomy doesn't replace senior engineers' intuition; it decomposes that intuition into verifiable questions, evidence, and rules so others can follow under pressure.

Measuring Whether the Classification System Is Actually Improving

Shipping classification fields ≠ done. Teams need outcome metrics to judge if it reduces incident loss and collaboration cost.

Watch Classification Quality, Not Just Fill Rate

Revision count isn't necessarily bad—early increases may mean the team starts honestly recording cognition changes. What must decrease is oscillation without new evidence, or wrong initial judgments persisting uncorrected.

Evaluate Errors with a Cost Matrix

Different misclassifications carry different costs. Misrouting a low-impact event to high priority mainly causes alert fatigue; misclassifying a high-impact security event as ordinary performance issue can be far more severe. Therefore models must not chase overall accuracy alone. Assign business-risk-based costs to error types and focus on:

Severe event missed

Cross-team misrouting

Treating secondary symptom as root cause

Triggering high-risk action at low confidence

Gray failures masked by global averages

Feed Postmortem Lessons Back Into Rules, Not Just Docs

Every postmortem should produce at least three feedback types:

New classification terms or boundary revisions

Earlier-detectable evidence and detection signals

New or adjusted classification-to-playbook mappings

If postmortem only adds a page to the incident doc, next time still relies on memory search. Pushing conclusions into vocabulary, event clustering, evidence rules, and playbooks is how experience truly enters the system.

From Individual Intuition to Organizational Capability

Small systems can rely on a few globals-aware people to hold stability. As scale grows, service boundaries, org boundaries, and regional boundaries multiply—no one can retain all details long-term. Fault classification's purpose is to establish a shared language for this complex collaboration.

This language doesn't aim to guess the unique root cause in minute one. It first answers verifiable questions: where did it fail, how did it fail, who is affected, how long has it persisted, how strong is the evidence, to what extent are business promises damaged. Answers evolve with the scene, but every change has a traceable basis.

A mature fault classification system preserves not who guessed right at the time, but how the organization used evidence to shrink uncertainty and bring correct actions forward.

From 100K to 1M QPS, teams can start by unifying fault vocabulary and incident templates. At 10M QPS, further investment in auto-clustering, multi-dimensional classification, confidence scoring, and policy routing is needed. Evolution focus isn't more complex algorithms, but ensuring every classification is explainable, every action reversible, and every postmortem improves the next judgment.

Back to that 02:17 AM alert storm: if your system only emits a "database fault" label, what category of evidence do you most wish it would supply next?