Fault Drills at 10M QPS: From Zero to Regular Cadence

The article opens with a typical first drill: taking down a cache replica, watching a red line on the dashboard, and declaring success after five minutes. The author argues this only proves the system didn't crash under a controlled action; it does not validate the full response chain — detection, decision, mitigation, recovery — nor does it test assumptions like automatic failover, on-call judgment, degradation authority, data consistency checks, or new-hire readiness at 3 AM.

Why Runbooks Still Fail in Real Incidents

Many teams have multi‑AZ deployments, backups, runbooks, and on‑call rotations, but these preparations remain static. Architecture documents describe design intent (e.g., two AZs, primary‑secondary DB, retryable messaging), while real incidents test runtime behavior: does traffic actually shift? Do connections rebuild? Does the old primary keep writing? Does backlog crush recovering downstream services? Runbooks prescribe an ideal sequence, but incidents rarely follow chapters; alerts may show DB timeouts while user complaints hit the payment page, some regions recover while others retry, and the person who owns a critical switch is unreachable. Personnel experience also expires: commands change, owners rotate, and new hires have never operated under failure conditions.

At 10 M QPS, small issues (a 200 ms dependency slowdown, a retry doubling load, a 30 s AZ switch) that were masked by capacity headroom now hit many requests simultaneously, and recovery actions themselves generate massive secondary load. The failure domain, concurrent actions, and state combinations explode; no single team can mentally simulate them all. Drills reduce unknowns through controlled, incremental experiments.

What a Real Drill Actually Tests

Fault injection is a technique; a drill verifies the entire response chain: from user impact detection, to on‑call severity assessment, to commander’s mitigation choice, to executor’s safe operation, to team confirmation of recovery, and finally feeding findings back into engineering improvements. A complete drill examines four dimensions:

System behavior: redundancy, isolation, rate‑limiting, degradation, failover, recovery — observing real business metrics, not just console “success”.

Observability: Do alerts fire before mass user complaints? Do they pinpoint affected user journeys and failure domains? Can dashboards support decisions? If you only see CPU and error logs but not order success rate, you’re guessing.

Process: Who declares the incident, who commands, who operates, who records the timeline, what triggers escalation or rollback — walked through under pressure to prevent simultaneous conflicting changes and repeated questions.

People: Can primary/secondary on‑call and dependent teams execute without the original author? The test is whether docs, permissions, and tools enable reliable actions by ordinary engineers.

The drill target is not the machine you shut down, but the end‑to‑end capability from anomaly signal to business recovery. Success criteria shift: automatic absorption is good, but if alerts and degradation weren’t validated the drill goal is unmet. Conversely, exposing a failover failure is valuable if safety boundaries held and remediation follows. The worst outcome is discovering nothing.

Draw a Risk Map Before Picking Scenarios

Teams often start with tool‑supported actions (kill process, add latency, partition network, saturate CPU). A better starting point is the business journey: list steps users take to achieve core goals, expand each step’s dependent services, data, network, and third parties, then ask three questions:

Where are single points or shared failure domains?

Which anomaly would breach existing isolation and affect the most users?

Which recovery action lacks recent evidence?

For a transaction chain, gateways and order services may appear multi‑cluster and multi‑replica, but deeper inspection might reveal shared config‑push channels, a shared cache cluster for inventory and marketing, and DB failover permission held by one person. The risk map surfaces these hidden couplings.

Scenarios are ranked by four dimensions: business impact, likelihood, knowledge gap, and drill risk. High scores on the first three warrant early validation; high drill risk calls for narrower scope or tabletop exercises first. Each scenario must state a falsifiable hypothesis, e.g., “After a full AZ partition, core order success rate recovers to target within 60 s, traffic does not flow back to the failed AZ, and no duplicate orders occur.” This defines fault, scope, expected behavior, recovery time, and correctness constraints. “Verify HA” is a poor hypothesis because any outcome can be rationalized.

Safety Boundaries Determine How Far a Drill Can Go

Drills aim for production realism without causing real incidents. Six boundaries must be defined:

Impact scope: test tenant, internal users, single API, single shard, or single region.

Traffic proportion: start with shadow traffic, then small real‑traffic percentages, gradually expanding.

Time window: avoid business peaks and high‑risk changes; set a maximum duration.

Business red lines: success rate, latency, data correctness, complaint volume thresholds that trigger immediate stop.

Technical red lines: connection counts, queue depths, replication lag, resource watermarks that trigger stop.

Rollback path: who executes, how long it takes, how to confirm system and data recovery.

Automate what can be automated: fault injection auto‑expiry, metric‑based auto‑stop, target whitelists, prevention of simultaneous correlated fault injections. Keep a manual kill switch, but it shouldn’t be the only guard. Blast radius expands in layers: tabletop → test env → shadow traffic → production small‑traffic → full fault‑domain drills only for teams with mature protections. The first step need not be “pull a data center”; pick a controllable scenario where the team isn’t fully certain of the outcome.

Game Day as an Observable Experiment

A mature Game Day follows a hypothesis‑driven, controlled experiment with stop conditions and a recorded timeline. Pre‑drill: freeze non‑essential changes, confirm roles, metrics, comms channels, permissions, rollback actions. The injector knows the exact fault; on‑call and responders may not, to test detection and diagnosis. Everyone must know red lines and who can abort.

During the drill, a scribe maintains a unified timeline recording not just “10:05 injected latency” but first business anomaly, first actionable alert, incident classification, role assumption, mitigation decision, execution result, recovery judgment, and data validation. Without a timeline, retrospectives devolve into memory‑based arguments.

Roles are kept simple and distinct:

Injector: executes and reverts the fault, does not speculate on root cause.

Primary on‑call: first response per real on‑call process.

Incident commander: sets severity, goals, decision tempo, cross‑team coordination; does not dive into logs.

Service owner: diagnosis and operations, states action risks.

Scribe: maintains factual timeline, prevents discussion from overwriting evidence.

Safety officer: independently watches red lines, can halt directly.

Small teams may combine roles, but the person who injects the fault should not also declare recovery. Injectors know the answer and may unintentionally hint; operators may mistake command success for user recovery. Hints are allowed but logged with time and reason; if on‑call gets stuck, the safety officer intervenes before risk grows — revealing gaps in docs, training, or observability.

Drill end requires explicit conditions: fault reverted, traffic normalized, error rate recovered, backlog controlled, data consistent, protection switches correct. Many real incidents’ second wave hits after “main fix”: caches refill simultaneously, consumers chase backlog at full speed, healthy instances crushed by returning traffic.

Scenario Library: From Component Failures to Business Failures

Component‑level faults (process kill, network latency, disk read‑only) are good starters — clear, automatable. After a few rounds, staying at component level creates a comfort zone: same alerts, same recovery path, missing real‑world combinations. Expand the library along three axes:

Scope: single instance → single shard → single AZ → single region → shared control plane. Expanding scope validates whether isolation boundaries truly exist.

Failure mode: not just hard down, but slow, wrong, stale, flaky. Intermittent timeouts, stale reads, partial errors, network jitter mimic nasty incidents that trigger retries, queuing, and bad decisions.

Recovery process: many drills test only fault injection, not recovery. Traffic cutback, cache rebuild, message catch‑up, data reconciliation, degradation exit can create new pressure.

Combined faults need caution. Simultaneously killing DB, messaging, and config center on day one just yields “everything broke.” Better: pick causally linked combos from real incidents or risk analysis. Example: AZ failover depends on config center — verify local fallback when control plane fails. Cache miss causes DB back‑source — verify DB protection holds. At 10 M QPS, multi‑region, multi‑cluster, heterogeneous infra, and shared control planes (routing, config, identity, monitoring) mean a local service fault may be less dangerous than a shared control‑plane outage that strips coordination across multiple failure domains. Scenario libraries must regularly test control‑plane failure modes and whether data planes can hold last known good state.

Don’t Hide Problems Behind “Drill Successful”

Post‑drill conclusions like “system met expectations” or “team responded promptly” carry no information without standardized metrics. Record metrics across five phases — detect, respond, control, recover, learn — without turning them into personal scorecards.

Detect: time from injection to first valid signal; time from signal to confirmed user impact. First alert may be unactionable (e.g., pod restart) — keep tracking until a decision‑supporting signal appears.

Respond: incident classification, commander in place, key teams engaged, first mitigation decision. Look for collaboration blockers, not individual rankings.

Control: peak impact, affected user percentage, error budget consumed, whether preset boundaries were breached. Reflects isolation and mitigation capability.

Recover: not just process restart. Track time from mitigation to business recovery, to data consistency, from fault removal to backlog drain. User‑ and data‑level tails must not be missed.

Learn: new risks discovered, remediation closed on schedule, regression passed. Many findings don’t mean a bad system; recurring same issue is the warning sign.

Illustrative example: 14:00 inject single‑AZ network partition; 14:01 infra alert; 14:04 on‑call confirms settlement success rate drop; 14:07 enable non‑core degradation; 14:10 core metrics recover; 14:25 data reconciliation done. Three improvement leads: business alert lagged infra alert by 3 min; impact assessment took 3 min; data validation took 15 min. These numbers are illustrative, not industry benchmarks. Each team sets targets around its SLO, RTO, RPO, and correctness requirements. Trend over successive drills (faster impact confirmation, fewer manual steps, more complete recovery evidence) shows capability growth.

Retrospective classifies outcomes into four types:

As expected — hypothesis supported within scope.

Protection worked but performance or recovery time missed target.

Unexpected coupling emerged — requires architecture or scenario model change.

Drill could not continue — due to permissions, tooling, docs, or insufficient safety boundaries.

Type 4 is common and valuable. If on‑call lacks DB failover permission, real incidents will stall too. Don’t borrow admin creds to “finish” the drill and report success.

From One‑Off Event to Stable Engineering Rhythm

Drills often start after leadership attention or a major incident. First run is well‑prepared, well‑attended, thick report. Months later, scenarios stale, remediation open, drills stop. To make them routine, embed into daily engineering cadence. Frequency shouldn’t be uniform: core paths, recent major changes, historical hotspots, and expired recovery evidence need higher frequency; low‑risk, stable, well‑covered areas can run less often. A starter rhythm:

Weekly: service teams run low‑risk, auto‑recoverable component drills.

Monthly: business domains run dependency or end‑to‑end journey drills, rotating primary on‑call.

Quarterly: cross‑team fault‑domain or recovery drills, validating command and capacity.

After major architecture changes, region expansions, or post‑incident remediation: targeted regression drills.

Adjust based on change velocity and risk. “Regular” ≠ mechanical repetition. Each drill should vary at least one variable: different on‑call, different region, added recovery phase, tighter target, or expanded verified boundary.

Remediation items must enter the existing work tracking system with owner, priority, deadline. Reports gathering dust become memorials. High‑risk issues should block related traffic scaling, similar changes, or drill intensity increases until closed.

Platform automation should be incremental. Early automation targets repetitive, error‑prone steps: target whitelists, approvals, fault auto‑revert, red‑line detection, timeline collection, metric snapshots, report templates. Scenario selection, business hypotheses, risk judgment remain human. Chasing a “chaos platform that covers all faults” early wastes effort on injection quantity.

Psychological safety is essential. Drills expose system and organizational design flaws, not individuals. Facilitators should state upfront: you can stop, ask for help, admit ignorance. Retrospectives focus on “what conditions made this error likely,” not “why didn’t you remember.” Only when remediation is closed and capability re‑verified does a drill truly end.

How Drill Focus Shifts with Scale

100k, 1M, and 10M QPS all need drills, but emphasis differs:

100k QPS: Few clusters, close team proximity. Risks: basic redundancy not working, incomplete runbooks, over‑reliance on core members. Drills cover single‑point failures, data recovery, basic on‑call flow.

1M QPS: Service count and dependencies grow; local faults propagate via retries, shared pools, message backlogs. Focus: dependency governance, business degradation, fault‑domain isolation, cross‑team response. Single‑service drills still needed but can’t replace full user journeys.

10M QPS: Multi‑region, multi‑cluster, heterogeneous infra, shared control planes explode state combinations. Traffic shift involves target cluster capacity, data replication, session state, cache refill, background backlog. Drill focus moves to fault‑domain capacity, control‑plane failure, combined faults, recovery storms, while automating continuous validation of mature low‑risk scenarios.

The difference isn’t running injections 10× more often. Larger scale means you can’t rely on one big all‑hands event. Mature approach runs two tracks in parallel: low‑risk scenarios in continuous automation (fresh evidence), high‑risk scenarios with rigorous design and layered expansion (exploring complex unknowns). Also track evidence freshness: topology, routing, data volume, owners, permissions change constantly. Last year’s successful region failover doesn’t prove this year’s. Tag key recovery capabilities with validation periods; expired ones become “unknown” and re‑enter the drill schedule.

Turn Unknowns into Next Verifiable Hypotheses

Moving from zero to regular is less about tool upgrades than changing how the team works. Initially, failure is a taboo topic; only real incidents test the system. First drill reveals mundane but critical issues: broken doc links, missing permissions, alerts that don’t show business impact, rollback scripts tied to one person’s laptop. These “non‑architectural” problems directly determine whether an incident converges quickly.

After several rounds, drills become routine. Low‑risk scenarios run automatically; business domains schedule Game Days; major remediation requires regression. The question shifts from “Is the system HA?” to “When was this capability last validated, under what scope and traffic, and is the evidence still valid?”

Reliability isn’t a permanent attribute; it’s a set of engineering judgments that need continuously refreshed evidence.

You don’t start with the most dangerous scenario. Pick one core user journey, find one real but controllable unknown, write the hypothesis, red lines, and rollback, invite the on‑call to follow the real process. After the drill, capture the top few remediations, close them, re‑test, then put the next drill on the calendar.

When drills become a fixed rhythm, the team gains not the confidence of “we’ve seen every failure,” but a more reliable capability: facing an unseen failure, they still know how to detect, control, recover, and learn.

If you only have 30 minutes for a drill next week, which assumption that’s been written off as “should be fine” would you most want to verify?