Structured Incident Response at 10M QPS: From Random to Process-Driven

Why "Everyone Is Busy" Still Fails to Stop Incidents

Many teams blame technical skill gaps for chaotic incident scenes. Postmortems often conclude "on-call engineer unfamiliar with system," "missing monitoring," or "should have prepared scripts." These issues exist but don't explain why bringing the most knowledgeable people together doesn't speed recovery.

The root cause: during an incident four types of work happen simultaneously:

1 Assess impact scope and severity

2 Execute mitigation actions (rate limiting, degradation, rollback, scaling)

3 Collect evidence and locate trigger conditions

4 Communicate progress to users, business, and management

Without explicit role separation, everyone gravitates to their comfort zone. Developers keep reading code, ops keep watching resources, business keeps asking for ETAs. Local actions make sense individually but can conflict: one person prepares a rollback while another scales up; one tries to preserve state while another restarts instances. The article emphasizes: Emergency response is a collaboration system that helps teams make reversible decisions under incomplete information. Its first goal is to control impact; root cause analysis comes second.

Why Random Firefighting Suddenly Breaks at Scale

At 100K QPS a single service owner can judge, act, and communicate. At 1M QPS you need on-call rotations, escalation paths, and basic runbooks. At 10M QPS failures rarely stay within one service; response must cross organizational boundaries.

Example: an ingress layer normally handling 8M QPS sees an 8% retry rate, adding 640K QPS that propagates through auth, config, cache, and DB proxy. Each layer's minor retries can cascade into multi-layer queuing. "Check if it's a network issue" is no longer a clear action directive.

Scale introduces four changes:

Failure Propagation Outpaces Human Discussion

Queue lengths, connection counts, and retry traffic can change per second. If the team spends ten minutes discussing ownership, escalation, or whether to degrade, system state has already shifted. Processes must pre-design the first few minutes' common actions so responders don't start from a blank slate.

Locally Correct Does Not Equal Globally Correct

Cache team increases origin pull ratio to fix cache miss anomalies; DB team sees connection spikes and tightens ingress. Both actions have merit individually, but together cause traffic oscillations. High-scale incidents need one person maintaining a global goal and sequencing conflicting actions.

More Participants Increase Communication Overhead

Five people can discuss freely in a voice room; twenty simultaneous speakers drown key information. Command, execution, recording, and external communication must be separated. Otherwise the technical lead constantly answers "how's it going?" and cannot think continuously; newcomers repeat already-confirmed facts.

Change Risk Is Amplified During Incidents

Normal releases have reviews, canary, and observation windows. Incident scenes skip these because everyone wants fast recovery. An "emergency fix" without a rollback plan can turn a single-region issue into a global outage. Process cannot eliminate risk but can force every high-risk action to answer three questions: expected effect, what metric to watch, how to revert on failure.

10M QPS doesn't demand more complex processes, but more deterministic ones. After an incident, which actions trigger automatically, which decisions require commander approval, and what information must enter the timeline should be predefined.

Using an Incident State Machine to Carry the Process

Some teams have dozens of pages of emergency policies yet scenes remain chaotic. The problem isn't document volume but lack of clear state mapping. Participants finish reading and still don't know the current step or exit criteria for the next.

A practical approach: model an incident as a state machine. Each state has a goal, inputs, exit conditions, and an owner.

Unconfirmed: Determine If This Is an Incident

Single-instance jitter, a false alert, and sustained user impact require different handling. On-call must complete minimum confirmation within an agreed time: is the alert real, does it affect internal metrics or user paths, is it expanding? Don't seek full root cause; if evidence warrants "organized response needed," move to Activated. Delayed activation usually costs more than a cautious escalation.

Activated: Organize People, Channels, and Goals

On activation, the system generates a unique incident ID, creates command channel, voice room, and timeline. First responder can act as Incident Commander but must explicitly declare the role, not assume "the senior person naturally commands."

Outputs at this stage should be concise:

Current severity and user impact

Known start time and affected zones

Incident Commander, Execution Lead, Communication Lead, Scribe

Next status update time

First batch of mitigation actions

Controlling: Stop Impact Expansion

Focus on boundaries: can the failure be confined to one region, tenant group, feature, or traffic slice? Common tactics: traffic switching, feature degradation, stopping non-critical jobs, freezing changes, isolating faulty dependencies.

Each action must record owner, start time, expected signal, and rollback condition. Commander doesn't execute but judges conflicts and limits concurrent changes.

Recovering & Observing: Recovery Isn't a Single Green Line

Core metrics returning to normal only means recovery started. Backlogged queues, cache rewarming, delayed tasks, and failed retries can cause a second shock. Before entering Observing, confirm user success rate, latency, error volume, and resource levels all recover together, not just the alert-triggering metric.

Observation window depends on system cycles. An online service with per-minute steady throughput may need 15 minutes; batch/settlement systems need a full cycle. If metrics degrade again, state returns to Controlling, not an unstructured re-discussion.

Closed & Pending Review: Closing Isn't the End

Before closure, preserve timeline, key queries, change records, and decision rationale. Action items must enter the regular task system with owners and deadlines. "Strengthen monitoring" is not an action item — it lacks object and verifiability.

The state machine turns emergency response from a chat log into an observable, auditable workflow.

Who Commands, Who Executes, Who Speaks Externally

Process adoption starts with role separation. One person can wear multiple hats in small incidents, but responsibilities cannot disappear.

The Incident Commander may not know all answers but must continuously ask the right questions:

1 What user impact is happening now?

2 Is impact still expanding?

3 What is the current top control objective?

4 Which actions are executing, and do they conflict?

5 When is the next decision point?

These five questions shift the scene from "what did everyone discover?" to "what will we change next?"

Single Command Channel, Multiple Technical Branches

Large incidents can spawn technical branches (DB, traffic, client) but conclusions must flow back to the command channel. Main channel keeps only facts, decisions, actions, results — no lengthy reasoning.

A practical message format:

02:21, Ingress Owner, plan to shift 10% of East China traffic to standby cluster. Expect payment success rate to recover within 3 minutes; if standby cluster CPU exceeds 65% or success rate doesn't improve, rollback. Incident Commander approved.

It adds little length over "I'll try shifting a bit" but preserves time, owner, action, expectation, and exit condition. Scribe can drop it into the timeline; newcomers instantly grasp what's happening.

Command Must Be Transferable

Long incidents degrade judgment via fatigue. Handoff cannot be just "you take over." It must include current state, known impact, actions in progress, unverified hypotheses, next decision point, and external commitments.

After handoff, new Commander confirms in main channel. Only one final decision-maker exists; previous Commander becomes advisor or rests. Dual command seems safer but creates two priority sets.

Severity Grading Matches Response Intensity

Grading determines how many people to wake, sync frequency, and what temporary permissions to enable. It's a tool to match response intensity, not a scary label.

Teams can use S0-S3 or P0-P3; naming doesn't matter. Criteria should rely on observable facts.

Judge breadth, depth, duration, and reversibility together. 5% error rate on login path may be more severe than 20% on non-core recommendations; even a few requests causing accounting corruption should raise severity.

Grading isn't one-time. Escalate when impact expands; de-escalate when stable. Escalation conditions should be pre-written: crossing second region, core success rate below threshold, exceeding observation window, or data integrity risk.

Good grading makes organizational cost rise with incident risk, not with chat-room emotion.

First 15 Minutes: What the Process Should Do for You

The first 15 minutes are most easily wasted. Information is scarcest, change fastest, and intuition drives parallel over-action. Teams can compress this window into a short outcome-oriented checklist, not a mechanical tick-box.

0-3 Minutes: Confirm User Impact

Check if alert stems from collection anomaly

Cross-verify with one end-to-end user metric

Mark first anomaly time, scope, trend

If activation criteria met, create incident immediately

3-7 Minutes: Establish Command & Shared Facts

Assign Incident Commander, Execution Lead, Communication Lead, Scribe

Announce severity and next update time

Freeze non-essential changes that could cloud judgment

Create a short "Known / Unknown / Verifying" table

7-15 Minutes: Choose Minimum Reversible Mitigation Actions

List up to three most likely impact-expansion mechanisms

Prioritize actions that shrink failure boundary

Each action gets a single owner and observation metric

Limit concurrent high-risk changes

If no improvement, rollback per pre-defined conditions and escalate response

"Minimum reversible" matters. Changing ten parameters at once may temporarily recover but obscures which action worked; if things worsen, clean rollback is impossible. Batched actions appear slower but reduce secondary incidents and repeated trial-and-error.

Runbooks should also center on these 15 minutes. Instead of a full system overview, provide executable entry points: how to view user impact, how to switch traffic, what degradation costs, when rollback is mandatory, which role to contact (not a fixed name).

One Trusted Timeline to Reduce Information Entropy

Incident scenes suffer information overload and version inconsistency. Screenshots scatter across groups; verbal conclusions lack sources; debunked hypotheses resurface minutes later.

Timeline should record six event types:

1 User impact and key metric changes

2 Severity changes and role changes

3 Approved actions and their execution results

4 Important hypotheses and supporting/contradicting evidence

5 External status publications and commitments

6 Recovery, observation, and closure decisions

Distinguish facts, hypotheses, and decisions when recording.

Fact: 02:18, payment success rate dropped from 99.95% to 91.2%

Hypothesis: ingress connection pool exhaustion may cause request queuing, comparing with healthy region

Decision: 02:24, approved shifting 10% East China traffic to standby cluster

Result: 02:28, success rate recovered to 96.8%, standby cluster resource levels normal

Mixing these four types in one sentence makes later reconstruction of decision context impossible. Separation enables postmortems to rebuild the reasoning, not just judge outcomes.

Fixed Cadence Beats Ad-Hoc Updates

Communication Lead updates status on a fixed cadence per severity, even if no progress: "Impact not expanding, two mitigations under observation, next update in 15 minutes." Fixed rhythm reduces business stakeholder interruptions, preserving technical focus time.

External communication only states confirmed facts:

Which functions and regions are affected

What workarounds users can apply

What the team is doing

Next update time

If root cause and ETA are unconfirmed, explicitly write "still investigating." Premature ETA commitments force risky actions around the promise and erode future trust.

Automate Transport First, Decision Support Later

Incident bots should first handle deterministic work: create channels, invite on-call roles, generate timeline, fetch release records, pin monitoring dashboards, remind status updates, archive chats and change logs.

Next stage: auto-aggregate information — correlate alerts to service topology, list changes around anomaly onset, compare healthy vs. unhealthy regions. Automation output must retain source and timestamp; never package correlation as causation.

Full automation of high-risk mitigations requires caution. Auto-pause obviously anomalous canary releases or shift small traffic per predefined thresholds; global degradation, data repair, or irreversible changes still need human approval and two-person verification.

How to Know the Process Actually Improved

Mean time to recovery (MTTR) alone misleads. Incident samples are few, severity varies; one complex outlier skews average. Better: break response into phases and measure speed, quality, and sustainability together.

Observe three quality signals:

Did conflicting actions occur during the incident?

How long until a newcomer understands current state?

Are action items closed on time and validated via drills?

Postmortems Examine Systems, Not Individuals

"Engineer X didn't escalate in time" is a symptom. Keep asking: were escalation criteria clear? Could on-call see user impact? Is the call-out process fast enough? Does escalation invite negative judgment?

Blameless postmortem doesn't dodge accountability. It demands stricter tracing of system conditions and decision rationale, avoiding hindsight bias. If avoiding an error requires a 10-year expert, the organization hasn't converted experience into tools, rules, or training.

Action items should fall into four categories:

1 Eliminate failure trigger conditions

2 Reduce failure blast radius

3 Accelerate detection and control

4 Improve response collaboration

Each needs a verification method. "Add user success rate alert for payment path, verify it fires within 2 minutes in a drill" is far more actionable than "improve monitoring."

Drills Are Integration Tests, Not Performances

Tabletop exercises validate grading, roles, decisions; fault injection validates monitoring, degradation, recovery; cross-team drills expose contact list, permission, and handoff gaps.

Drill scenarios must not give answers upfront; only provide facts as they emerge. Observe whether they activate per process, whether someone maintains global view, whether actions and exit conditions are recorded. Update runbooks after drill, re-validate — only then has the process entered production.

An un-exercised emergency process is like an untested disaster recovery script — just a hypothesis.

Evolving from Hero Firefighting to Organizational Capability

Process adoption doesn't mean building a massive incident management platform day one. Teams should evolve based on actual incident frequency and organizational complexity.

Phase 1: Make Response Visible

Unify incident entry, grading rules, on-call schedules, timeline templates. Goal: every incident answers who owns it, what's the impact, what actions were taken, when did it recover.

Phase 2: Make Actions Repeatable

Build concise runbooks for frequent failures, turning rate limiting, degradation, rollback, traffic shifting into verifiable, reversible standard operations. Runbooks evolve with system changes and are pruned via drills.

Phase 3: Make Collaboration Auto-Start

Use bots to create channels, summon roles, attach monitoring and changes, remind status updates. Automation reduces toil but doesn't replace commander's risk judgment.

Phase 4: Make Organization Learn from Incidents

Close the loop: incidents, postmortems, action items, drills. Recurring incidents aren't just about same root cause; ask why the same control capability wasn't reinforced. Management shifts focus from "who fixed it this time" to "can a regular on-call engineer handle it next time."

Process must allow exceptions. Novel failures won't fit the playbook; commander can deviate but must document reason and outcome. Mature process doesn't turn people into script executors; it pre-eliminates low-value choices so attention stays on the truly unknown.

Move the Next 2 AM Incident to Daytime Preparation

Random firefighting relies on luck: the on-call engineer happens to know the system, the expert happens to be online, the first attempt happens to work. Process-driven response acknowledges information is always incomplete during incidents, so it pre-designs states, roles, grading, timeline, and exit conditions.

As systems grow from 100K to 10M QPS, technical chains lengthen — controlled by isolation, redundancy, rate limiting. Organizational chains also lengthen — controlled by unified command, clear accountability, and steady rhythm. Missing either chain amplifies failures.

Reliable emergency capability should let an ordinary on-call engineer make good-enough decisions with process support.

Review your team's last incident: if you swapped the most senior person with someone two months into rotation, would your current process let them establish command, control impact, and accurately pass information within 15 minutes?