Incident Command: Who Speaks, Decides & Communicates Externally?

1. The 98-Minute Case Study: Four People Doing Right Things, Organizationally Chaotic

An e-commerce platform's Thursday evening promotion triggered an alert at 20:02: order API P99 latency rose from 220ms to 3s, error rate 7%. Over 98 minutes, 30 people spoke in the chat, eight windows changed configuration, the same status was repeatedly asked, and some monitored on phones while commuting. No one announced start or end, and no one knew who had final authority.

The timeline shows four engineers each taking technically correct actions:

On-call A scaled order service from 40 to 80 replicas (standard traffic response).

Owner B shortened client timeout from 3s to 800ms (fast-fail to protect thread pools).

Infra C noticed DB connections at 88% and suggested rolling back the latest release.

On-call D executed a rollback but mistakenly rolled back the gateway service, not the order service, because he assumed C meant the gateway release.

Later, cache engineer E cleared hot keys and P99 dropped to 600ms. Despite each action being defensible in isolation, the incident suffered from:

Four directions running simultaneously (scale, config change, rollback, cache clear) with no shared system state record.

One suggestion mis-executed, discovered 30 minutes later.

User complaints arrived at 20:38; no one answered whether to issue a public announcement.

Payment team investigated independently because they didn't know someone was managing the order chain.

No defined end moment; the chat simply went quiet.

Only ~20 minutes were spent on technical operations; the remaining 78 minutes were consumed by mutual confirmation, repeated judgment, waiting, wrong directions, and status polling.

2. Decision Throughput: The Real Bottleneck

2.1 A Simple Model

Treat an incident as a decision pipeline: T ≈ n × t + technical_fix_time, where n is the number of decisions (typically 8–15) and t is average minutes per decision (5–15). Decision queuing alone consumes 40–150 minutes while technical fix may need only 10 minutes.

Three reasons for high t (none due to incompetence):

Unknown decision authority – anyone can propose, no one dares decide; proposals wait for owner, who may be offline.

Unknown decisions already made – A thinks B is rolling back, B thinks A is scaling; duplicated or cancelling actions.

Unknown priority – five things to do, five people pick one each, no sequencing; the 98-minute case exemplifies four parallel directions none completed properly.

2.2 What Incident Management Must Solve

Not "make engineers smarter" but:

Compress t to ≤2 minutes via explicit role authorization (pre-defined, no ad-hoc meetings).

Compress n to necessary minimum by codifying proven decisions into runbooks and playbooks, turning them into reflexes.

These are the core of Incident Command (IC).

2.3 IC Is Not the Strongest Technician

Command and technical troubleshooting require different cognitive modes. Troubleshooting demands deep, uninterrupted focus; command demands constant interruption, information aggregation, and rapid small decisions. Doing both leads to failure in both. Rule: IC avoids keyboard; once IC starts troubleshooting, they cease to be IC. If team is too small, note "IC missing" in postmortem as a gap, not blame.

3. Three Core Roles Plus One Optional

3.1 Role Comparison

IC (Incident Commander) – decides what to do, what not to do, when to stop. Suited for authorized person who can say no and avoid details. Does not troubleshoot, run commands, or write code.

OL (Ops Lead) – translates IC decisions into operations, leads technical investigation. Best: person who knows the system best (usually service owner). Does not communicate externally or set priorities.

CL (Comms Lead) – broadcasts internal status, issues external announcements, shields from irrelevant queries. Suited for clear communicator under pressure (PM, duty manager, TAM). Does not judge technology or change systems.

Scribe (optional) – records timeline and decision log for postmortem. Can be anyone who types, or automated. Does not participate in decisions.

Only IC must be a long-term assignment; OL and CL can hand over during incident; Scribe can be automated.

3.2 IC's Five Actions

Opening (first 3 minutes) : declare incident, self-identify as IC, give one-line status, set severity, assign OL and CL.

Collecting (continuous) : all information flows to IC, all decisions flow from IC. Every proposal gets explicit response: "go", "no, do X first", or "defer, review in 5 min". Silence is the costliest response.

Prioritizing (continuous) : when multiple actions possible, IC says "only do this one".

Timekeeping (continuous) : set next status update (usually 15 min), must update even if no progress.

Closing : declare mitigation, declare end, assign postmortem owner, initiate postmortem.

Actions 2 and 4 are most often skipped yet most critical.

3.3 IC Opening Template (Copy-Paste Ready)

🚨 Incident Start · INC-2026-1106-01

I am IC: @ZhangSan (I decide for this incident)
Severity: P1 (per order success rate 7%, see severity tree)
Symptom: order API P99 220ms → 3s, error rate 0.2% → 7%, started 20:02
Impact: order main chain, estimated 40% requests affected, no data corruption signs
Current judgment: unknown, verifying if related to 19:50 release

Roles:
- IC: @ZhangSan
- Ops Lead: @LiSi (technical investigation & operations for order chain)
- Comms Lead: @WangWu (internal/external comms, all external messages from her)

Rhythm:
- Status update every 15 minutes, next at 20:22
- Any operation must announce in group "I will do X, est. Y minutes" and get my confirmation
- Cross-team help: contact @WangWu, do not @all

Incident group: this group
Dashboard: https://grafana.internal/d/incident-order

Each line serves a purpose: "I am IC" eliminates authority ambiguity; severity determines notification scope and who shows up; symptom+impact gives common starting point; roles clarify who to contact (especially CL to shield IC); rhythm converts anxiety into scheduled waiting; pre-action announcement prevents parallel conflicting actions (cost 5 seconds vs 30 minutes for mis-rollback).

3.4 Handoff: 24-Hour Incidents Need Shift Changes

Handoff message answers four questions:

🔄 IC Handoff · INC-2026-1106-01 · 20:00

1. Current system state (verified facts, no speculation)
   Order service rolled back to v2.31.4, error rate 7% → 0.4%, P99 still 1.2s (baseline 220ms)
2. Excluded directions (prevent rework)
   DB connections normal (62%), cache hit rate recovered 94%, network no packet loss
3. Current leading hypothesis + next verification
   Suspect slow query in old version v2.31.4 causing high P99, next: compare SQL execution plans
4. Pending decisions (hanging)
   - Whether to rate-limit order entry (I lean yes, not decided)
   - Whether to send second external announcement (first sent 20:40)

Item 2 (excluded directions) is highest value and most often omitted; without it, incoming IC repeats dead ends. Item 4 prevents decisions silently disappearing after shift change.

3.5 Counter-Intuitive Principle: IC Does Not Chase "Fastest Recovery"

IC optimizes for overall incident cost, not speed. Example: bleeding stopped (error rate back to baseline) but root cause unknown. Two choices: declare end and sleep, or spend 20 more minutes confirming root cause and writing announcement. Good IC chooses latter because:

Root-cause-unknown incidents have ~30% recurrence within 2 hours; recurrence recovery takes 1.5× longer as context is lost.

Vague external announcements cause support overload next day.

Hence IC needs an explicit "closure checklist" (see Section 9), not "looks okay".

4. Severity Classification: Not by Feeling

Alert severity (P0-P4) and incident severity (SEV-1 to SEV-4) are different. Alert severity is static per rule; incident severity is dynamic, set by IC on scene, can upgrade/downgrade. Alert severity drives notification channels and response SLA; incident severity drives who must join, external messaging, and postmortem level.

4.1 Two Dimensions Only

Incident severity uses only impact scope (how many users, how deep, data consequences) and trend (worsening, stable, improving). Must be decided in 3 minutes.

SEV-1 : core chain completely unavailable, data loss/corruption, or financial loss. Requires IC+OL+CL+business owner+tech lead; preliminary postmortem within 24h.

SEV-2 : core chain partially degraded, or fallback exists but not active. Requires IC+OL+CL; postmortem within 48h.

SEV-3 : non-core service degraded, or user can bypass. Requires IC+OL; postmortem within week (can batch).

SEV-4 : no user perception, only internal metric anomaly. No IC activation; handle as alert.

Two rules: (1) Err on higher severity; downgrade must leave audit trail (e.g.,

severity_change: SEV-2 → SEV-3 at 21:10 by @ZhangSan, reason: impact confirmed 3% with bypass

). Undocumented downgrades look like cover-ups. (2) SEV-1/SEV-2 boundary is data consequence, not severity level. Any data corruption or financial loss forces SEV-1 because it triggers non-technical workflows (reconciliation, compensation, legal, support).

4.2 Severity Drives Notification Matrix

Automate notifications so IC doesn't manually pull people. Table defines immediate, 15-min, 30-min auto-escalation per severity. Three engineering details: IC pool rotates like on-call; "15-min no mitigation" uses timer not human memory; notifications must carry context (incident ID, one-line symptom, group link, current IC) so awakened person joins group in 30 seconds.

4.3 Decision Tree for 3-Minute Classification

① Data loss/corruption, financial loss, or security breach?
   └─ Yes → SEV-1 (regardless of technical recovery)
   └─ No → ②
② Core chain (order/pay/login) fully unavailable & >5% users affected?
   └─ Yes → SEV-1
   └─ No → ③
③ Core chain partially degraded (success rate <95% or P99 >3× baseline) or >1% users affected?
   └─ Yes → SEV-2
   └─ No → ④
④ Non-core service degraded or core chain minor but bypassable?
   └─ Yes → SEV-3
   └─ No → SEV-4
⑤ After classification, ask: Is trend worsening?
   └─ Yes → upgrade one level (max SEV-1)
   └─ No → hold

Step ⑤ is often missed. Trend matters more than absolute value: 3% error rate doubling every 5 minutes is more dangerous than stable 10%. Re-evaluate at 15 minutes because criteria shift from "how bad" to "still worsening?".

5. IC's First 20 Minutes: Minute-by-Minute

5.1 Minutes 0-3: Verify & Declare

20:02  Receive alert. Do 30-second verification (metrics, pods, events, logs).
20:03  Confirm real. Do NOT deep-dive cause (that's OL).
       Do three things:
       1) Estimate impact (order success rate, downstream errors, support tickets)
       2) Classify via severity tree
       3) Post opening message (Section 3.3) in incident group
20:05  Opening message posted, pinned. You are now IC.

Trap: Don't wait for root cause to declare. "Current judgment: unknown" is acceptable. Delaying declaration 10 minutes leaves everyone fighting alone.

5.2 Minutes 3-10: Assign First Direction

20:05  Give OL a concrete first task, not "go investigate".
       ✅ "LiSi, spend 3 minutes confirming: was 19:50 release fully rolled out? DB connections? Downstream error rates? Report back."
       ❌ "LiSi, investigate cause." (He'll start from top, take 20 minutes)
20:08  CL in place, have her send internal broadcast (not external): "Owner assigned, investigating." Purpose: calm anxiety, not technical detail.
20:10  Group starts @-ing you. Deflect to CL: "All progress in pinned message, questions to @WangWu." Send once.

Key: Turn "open-ended investigation" into "verify a hypothesis". Open-ended averages 20-40 minutes; hypothesis verification 3-5 minutes.

5.3 Minutes 10-20: Make First Decision

20:15  OL reports: DB connections 88%, 19:50 release was C2, fully rolled out 12 minutes ago.
20:16  IC decides: rollback order service (not gateway), because connections are only worsening resource and release just completed. Explicit instruction:
       "LiSi, rollback order service to v2.31.3 using rollback-point checklist, est. 4 minutes. Announce in group before start."
20:17  Explicitly reject other two directions with reasons (critical step):
       - "Stop scaling, already 80 replicas, bottleneck not compute."
       - "Hold cache, re-evaluate after rollback if P99 still high."
20:20  Status update #2:
       "Decided rollback order service, started 20:17, est. complete 20:21. Error rate 9%, still rising.
       Next update 20:35 unless change."

Explicit rejection with reasons is most counter-intuitive yet vital. Unrejected directions don't vanish; they run in background. The 98-minute case had four parallel directions because no one said "stop this".

5.4 IC's Three Universal Phrases

Proposal not ready: "Received, park it, we'll evaluate after X finishes." (Gives concrete re-evaluation time, not "later".)

Unauthorized config change: "Who changed gateway config after 20:15? Tell me what changed now." (Ask facts, not blame; accountability in postmortem.)

Repeated "how is it?": "Check pinned. Next update 20:35." (Single reference, no repeat answers.)

Plus one for OL: "If no progress in 5 minutes, come back, we'll switch direction." Authorizes OL to abandon a path; without it, OL may grind 40 minutes fearing "can't say not done".

6. Decision Authority: Who Can Hit Rollback

Technical judgment ("should we rollback?") is separate from authority ("who can press the button"). Real dilemma: rollback may lose written data or interrupt ongoing activity. Executor hesitates, asks leader, leader says "you decide", 20 minutes lost.

6.1 Decision Authority Matrix

Pre-define for each decision type: who can decide alone, who needs two-party confirmation, who must escalate. Signed by tech lead, not just SRE.

Solo authority (reversible, bounded blast radius): scale replicas, enable pre-defined degradation switches, rollback C1/C2 changes. Mistakes quickly reversible, no waiting.

Two-party confirmation (affects others): rollback C3 (schema changes), rate limiting (business impact).

Mandatory escalation (irreversible or external consequences): C4 rollback, stop settlement jobs, external announcements.

6.2 Automate the Matrix

Don't leave matrix in Confluence. Embed in release/incident tool. Example script incident-decide.sh reads change class from pipeline, determines required approvers, sends approval requests via bot. Solves not "approval" but "who to ask" without on-scene thinking. Cost of figuring "who to ask" is 10 minutes; cost of wrong decision is 30 minutes.

6.3 Overriding Authority with Immunity

Must have someone with "overriding authority" during SEV-1/2: can stop any non-incident change/release/batch job/DDL, close or rate-limit any entry including revenue entries, reject any non-incident operation, demand 5-minute response in incident group. These decisions need no prior approval but must be logged in group and reviewed in postmortem. Accountability only for "no record", not "decision correctness". Authority without immunity equals no authority.

7. Escalation: Cross-Team Without Face Loss

Most time-consuming cross-team action is not technical collaboration but convincing the other team this concerns them . Failed pattern: @DBA "order slow, check DB" → "DB monitoring normal" → 15 minutes later "I looked, fine". Request shape was wrong: no specific metrics, no time window, no concrete ask, no receipt requirement.

7.1 Four-Part Escalation Request

【Escalation】INC-2026-1106-01 · SRE → DBA

1. Facts (not judgment)
   Since 20:02, order DB threads_running 12→180, threads_connected 62%→88%, same period order-api P99 220ms→3s.
2. What we've done (avoid duplicate work)
   Scaled order service to 80 replicas (no effect), rolled back v2.31.3 (error rate down but P99 not recovered).
3. What we need you to do (specific, single, with acceptance criteria)
   Confirm if any abnormal slow queries 20:00-20:20, and whether 19:50 release's new SQL caused full table scans. Acceptance: provide top5 slow queries with execution plans.
4. Deadline & receipt
   Reply "looking" or "not ours" by 20:40 (10 minutes). If no conclusion by 20:45, I'll treat this path as suspect and escalate to duty manager.

Part 1 uses facts (indisputable) not judgments (trigger defensiveness). Part 2 proves effort, reduces resistance. Part 3 demands single task with clear done criteria; vague multi-task requests get cherry-picked. Part 4 sets deadline, receipt, and explicit consequence of timeout.

7.2 Three Anti-Patterns

Judgment as fact : "Your DB dragged us down" → defense mode, 20 minutes proving innocence. Use facts, work starts in 10 seconds.

Five asks at once : "Check slow queries, connections, locks, replication lag, disk IO" → they pick easiest, reply "normal". Ask one thing, get result, ask next.

@all : Looks like notification, actually diffuses responsibility. Always @ specific person or on-call bot.

7.3 Escalation Triggers (Not Gut Feeling)

15 minutes no mitigation from IC declaration (pure timer).

Cross-service/team suspicion – escalate immediately, don't wait for confirmation; run in parallel with own investigation.

Irreversible decision needed (C4 rollback, stop business, drop requests) – escalate early, not at moment of action.

Second trigger is key: habit of "verify it's their problem before bothering them" costs 15-30 minutes. Correct: "I suspect, escalating now while I keep digging."

8. External Communication: Four Status Page Templates

Internal comms via group/pinned; external only via CL using fixed templates. Ad-hoc wording in incident produces three problems: speculation as fact, vague timestamps, no next-update commitment.

Three bottom lines: (1) Only verified facts and committed actions, no speculation. (2) Must have three times: publish time, impact window, next update time. (3) No generic apologies or technical details; users want "can I use it?", "when fixed?", "is my money safe?".

8.1 Four Templates

Template A: Incident Ongoing (first announcement, impact confirmed)

[Incident Notice] Some users may fail to place orders (updated 20:40)

Scope: Since 20:02, some users may encounter order failures or timeouts, affecting ~40% of order requests.
Payment unaffected; paid-but-incomplete orders will auto-compensate after recovery.

We are doing: rollback completed, error rate declining, confirming root cause.

Next update: 21:10 (or immediately on status change)

Template B: Incident Ongoing (progress update, 15-min cadence)

[Progress Update 21:10] Order success rate recovered to 96%

Changes since 20:40: rollback done, error rate 7% → 0.8%;
Order latency still ~3× normal, addressing.

User advice: if order fails, retry later, do not resubmit.
Data impact: no order data anomalies, paid orders unaffected.

Next update: 21:40 (or immediately on status change)

Template C: Mitigated (recovery announcement)

[Resolved 21:40] Order service restored

Impact window: 20:02–21:40, 98 minutes total.
Impact: ~40% order requests failed during peak; payment unaffected.
Current status: order success rate 99.9%, latency back to normal.

Follow-up: postmortem within 3 business days, per-order reconciliation for affected orders.
Contact support for questions.

Template D: SEV-1 Customer-Specific (one-to-one, not broadcast)

[Customer Notice · Send Individually]

Dear XX:
Tonight 20:02–21:40 our order service had an incident; your orders in this window may have failed.
We have identified and fixed; service now normal.

Action needed: please reconcile order count and fund flow for this window; reply to this email if discrepancies.
Dedicated contact (XXX, phone XXX) available 24h for any issues.
Formal postmortem within 3 business days.

Template D exists because SEV-1 customer comms cannot be broadcast. Broadcast addresses "all users"; the few key customers (usually <20) need reconciliation and compensation paths, not apologies.

8.2 CL as Shield, Not Just Announcer

Without CL, tech lead gets @-ed 8 times per 5 minutes (6 are "status?", "announce?", "support what to say?"). With CL, those 8 go to CL; tech lead gets one: "Status is XX, external message sent per script, you continue." CL selection: not "who's free" but "who withstands pressure and can say 'I cannot answer that now'". PM, duty manager, TAM, support lead all work if authorized for external speech.

8.3 Often Missed Audience: Customer Support

In SEV-1/2, first user contact is support, not engineers. If support doesn't know situation, they give inconsistent explanations, screenshots spread, creating second crisis. CL must send support talking points before first external announcement:

[Support Talking Points · Effective 20:45]

Current: order service incident, recovering, ETA 21:40.
Unified reply:
1. Acknowledge: "Yes, order service has an incident, engineers handling."
2. Give time: "Expected recovery within 40 minutes."
3. Give action: "If paid but order not created, auto-reorder after recovery, no double payment."
4. No promises: "Do not trust non-official channels."
Don't say: speculate cause, promise compensation, give exact recovery time (only "within 40 minutes").
Escalate: fund discrepancies, compensation demands → @WangWu.

Value: reduces engineer-support interruptions. One SEV-1 saw engineers interrupted 20 times explaining same thing; time lost exceeded technical investigation.

9. Closure: How to Know Incident Is Truly Over

Case study ended with "group gradually quiet" – nobody knows it's over, next day someone asks "how'd last night go?" wasting 30 minutes explaining. Closure needs explicit action, criteria, and announcer.

9.1 Four Simultaneous Conditions

SLI back to baseline band (not "better"): error rate <1.5× baseline, P99 <2× baseline, sustained 15 minutes.

All changed configs/versions recorded and current running version known. Prevents "half-rolled-back" state.

No pending irreversible ops (data fixes, reorders, reconciliation tasks started).

External announcements converged (final announcement sent, support talking points updated).

Condition 2 most ignored, highest cost: ad-hoc timeout changes, temporary health check removals, temporary rate limit relaxations left in system become next incident's landmines.

9.2 Structured Incident Record Schema (YAML)

Output at closure is not a document but a machine-readable record (feeds metrics, runbooks, training). Example:

id: INC-2026-1106-01
severity: SEV-2
severity_changes:
- {at: "2026-11-06T20:05:00+08:00", from: null, to: SEV-2, by: zhang_san}
- {at: "2026-11-06T20:20:00+08:00", from: SEV-2, to: SEV-1, by: zhang_san, reason: "payment success rate dropped, confirmed cross-chain"}
roles:
  ic: [zhang_san]
  ol: [li_si, zhao_liu]  # with handoffs in time order
  cl: [wang_wu]
  scribe: incident-bot  # automated
timeline:
- {at: "20:02", event: "alert_fired", source: alertmanager, detail: "OrderApiHighLatency"}
- {at: "20:05", event: "incident_declared", by: zhang_san}
- {at: "20:17", event: "action_started", by: li_si, detail: "rollback order-api v2.31.3"}
- {at: "20:24", event: "action_started", by: zhao_liu, detail: "rollback gateway", note: "inconsistent with IC directive, discovered 20:47"}
- {at: "21:40", event: "incident_resolved", by: zhang_san}
decisions:
- {at: "20:16", decision: "rollback order service not scale", by: zhang_san, rejected: ["scale to 120 replicas", "clear cache"]}
detection: {alert_to_ack: 60, ack_to_declare: 120, declare_to_first_action: 660}
escalations:
- {at: "20:22", from: sre, to: dba, reason: "connection anomaly", response_s: 540}
user_impact:
  duration_min: 98
  estimated_affected_requests: 41200
  data_loss: false
  financial_impact: false
actions_outstanding:
- {item: "restore temporarily relaxed gateway rate limit", owner: li_si, due: "2026-11-07"}
- {item: "reconcile 20:02-21:40 order vs payment flow diff", owner: fin_interface, due: "2026-11-08"}

Key fields: severity_changes (every change with reason, CI-enforced); timeline.note (e.g., "inconsistent with IC directive" – most valuable for postmortem); decisions.rejected (records what was vetoed, more informative than what was done); detection three intervals (feed Section 10 metrics); actions_outstanding (incident end ≠ work done).

9.3 Who Writes the Record

Not humans. Chat messages are timestamped; bot auto-extracts alert_fired, incident_declared, action_started, escalation. Humans only fill decisions, severity_changes, user_impact. Requires IC to use fixed phrases in chat (Section 3.2) – two sides of same coin.

10. MTTR Is Not One Number, It's Four

Teams report MTTR as average, but average lies in incidents. Break incident into four phases:

Failure injection → [MTTD] → Someone knows → [MTTA] → Someone on scene → [MTTK] → Bleeding stopped → [MTTR-R] → Recovered
   t0                    t1                    t2                    t3                    t4

MTTD = t1-t0  Detection: failure to someone alerted
MTTA = t2-t1  Acknowledgment: alerted to hands-on
MTTK = t3-t2  Kill (mitigation): hands-on to impact stops worsening
MTTR-R = t4-t3 Recovery: mitigation to full recovery
MTTR = t4-t0

Each phase maps to distinct improvements; mixing them hides what to fix.

10.1 Phase Ownership

MTTD: observability (alert coverage, latency, burn rate, human-reported).

MTTA: on-call system (notification routing, phone reach, runbook one-click).

MTTK: engineering + organization (degradation switches, rollback points, decision authority).

MTTR-R: architecture (data replay, cache rebuild, downstream catch-up).

10.2 Key Ratio: MTTK Share of MTTR

Healthy incident (60 min total) roughly: MTTD <5 min, MTTA <3 min, MTTK <20 min, MTTR-R remainder. If MTTK >50% of MTTR (common 60-80%), problem is almost certainly decision queuing, not technical – exactly the Section 2 model.

10.3 Compute from Records (Python)

#!/usr/bin/env python3
# incidents/metrics.py – compute MTTD/MTTA/MTTK/MTTR-R from incident records
# Usage: monthly postmortem & quarterly trends, output grouped by severity (not mean)

import json, pathlib, statistics
from collections import defaultdict

def load_incidents(root="incidents"):
    for p in pathlib.Path(root).rglob("*.json"):
        yield json.loads(p.read_text())

def phase_seconds(inc):
    d = inc["detection"]
    return {
        "mttd": d["alert_to_ack"],
        "mtta": d["ack_to_declare"],
        "mttk": d["declare_to_first_action"],
        "recovery": (inc["user_impact"]["duration_min"] * 60)
                      - d["alert_to_ack"] - d["ack_to_declare"] - d["declare_to_first_action"],
    }

def summarize(root="incidents"):
    buckets = defaultdict(lambda: defaultdict(list))
    for inc in load_incidents(root):
        ph = phase_seconds(inc)
        sev = inc["severity"]
        for k, v in ph.items():
            buckets[sev][k].append(v)
        buckets[sev]["mttk_share"].append(ph["mttk"] / max(sum(ph.values()), 1))
    for sev in sorted(buckets):
        print(f"
=== {sev} ===")
        for k, vals in buckets[sev].items():
            vals = sorted(vals)
            p50 = statistics.median(vals)
            p90 = vals[int(len(vals)*0.9)-1] if len(vals)>1 else vals[0]
            print(f"{k:12s} n={len(vals):3d}  P50={p50:7.1f}  P90={p90:7.1f}")

if __name__ == "__main__":
    summarize()

Two disciplines: (1) Group by severity, report P50/P90, not mean (SEV-1 may have 3/year, mean statistically meaningless). (2) Trend > absolute value: MTTK dropping from 22 to 14 minutes is more persuasive than "current MTTK is 14".

10.4 Never Use MTTR for Individual Performance

If MTTR enters personal KPIs, three inevitable outcomes: (1) Incidents stop being recorded (borderline events handled quietly, patterns reappear larger later). (2) Mitigation becomes aggressive ("restart and see") destroying diagnosability; rollback "succeeds" but failure persists. (3) Root cause analysis shallow because postmortem takes time, better to close ticket. Correct use: MTTR finds improvement targets, not evaluate people. Each quarter, identify longest phase, improve only that – more effective than everyone staring at total time.

11. Metrics: Five Numbers to Judge Incident Management

Beyond MTTK, need metrics measuring organizational performance in incidents – harder to game than time.

IC Readiness Rate : incidents with explicit IC declared / total incidents. Target >95% (SEV-1/2 100%). Easiest action to miss, prerequisite for all else.

Decision Latency : P90 from "proposal made" to "IC responds". Target <3 min. Directly measures decision throughput, no technical noise.

Parallel Operations Count : concurrent unconfirmed actions at any moment. Target =0. Prevents Section 1's four-way chaos.

Escalation Response Time : P50 from cross-team escalation to first receipt. Target <10 min. Measures real org collaboration, not documented process.

Recurring Incident Rate : incidents sharing root cause tag with past 12 months / total incidents. Target <10%. Only metric reflecting "postmortem actually changed something".

Fifth metric most powerful. Auto-implement: postmortem has root_cause_tag; new incident closure matches tags, auto-labels recurring: true. Once tracked, postmortem quality self-improves – authors think "will this tag reappear in six months?".