From Wiki Docs to Executable Incident Plans: Cutting MTTR from Hours to Minutes

At 2 a.m. a downstream risk‑control service timed out, waking an on‑call engineer who followed a wiki page titled “Big‑Sale Emergency Plan V3”. The steps required logging into a jump host, locating a configuration server, and flipping a feature flag from true to false. The documented machine had been decommissioned three months earlier and the flag name had changed, causing a 26‑minute recovery time. In a system handling tens of millions of QPS, that delay impacts billions of requests.

What is an incident plan and why it matters

An incident plan (pre‑plan) is a predefined set of actions for foreseeable failure scenarios such as downstream dependency outages, traffic spikes, data‑center failures, cache avalanches, or critical‑path latency. The goal is to decide the handling steps in advance, write them down, and eventually codify them into executable steps so that during a real incident the team follows the plan instead of improvising.

The five sins of document‑based plans

Staleness : Docs do not update automatically as services evolve, leading to expired procedures.

Manual execution : Human operators must copy commands; a typo or missed step can worsen the incident.

Untestability : Docs cannot be rehearsed; their effectiveness is unknown until a real failure.

Knowledge silos : Only the author and a few veterans know where the plan lives and how to use it.

Uncontrolled high‑risk actions : Dangerous operations (e.g., traffic cut‑off) lack permission checks and audit trails.

These problems become critical when the system scales to tens of millions of QPS because the cost of a slow recovery grows dramatically.

Why MTTR is bottlenecked

Fault handling consists of four stages: detection, diagnosis, decision, and execution. Monitoring compresses detection; tracing and logging compress diagnosis. In a document‑centric workflow, decision (searching the right plan) and execution (manual steps) dominate MTTR, especially at massive scale.

Layer 1: Structured, executable workflows

The first step is to transform natural‑language steps into a directed‑acyclic‑graph of actions. An action is a concrete API call, e.g., “adjust rate‑limit to 5000”. A full plan becomes a series of such actions bound to real control‑plane APIs. Two design rules are essential:

Idempotency : Re‑executing an action must not change the outcome, protecting against retries or duplicate clicks.

Reversibility : Every action should have an inverse (e.g., “restore original rate‑limit”) so that a failed step can be rolled back safely.

Layer 2: One‑click execution and gray‑scale rollback

With a structured DAG, a UI can trigger the entire plan with a single click. The system automatically orchestrates actions and shows progress, shrinking a 26‑minute manual process to seconds. However, rapid execution demands safety mechanisms:

Gray‑scale rollout : Instead of applying a change to 100 % of traffic, the system applies it incrementally (e.g., 10 %, then 30 %, then 100 %) with checkpoints that monitor latency, error rate, and success metrics.

Rollback : If a checkpoint detects metric degradation, the system aborts further steps and invokes the inverse actions automatically.

Layer 3: Permission, approval, and audit

When plans become powerful, access control is required. Low‑risk actions (e.g., non‑critical rate‑limit adjustments) may be executed by any on‑call engineer, while high‑impact actions (e.g., cutting an entire data‑center) require supervisory approval. Every execution is logged with who, when, which plan, parameters, per‑action results, and final outcome, supporting post‑mortem analysis and regulatory compliance.

Layer 4: Regular chaos‑engineered rehearsals

Even a system‑based plan can become stale as APIs disappear or topologies change. Regular rehearsals in a controlled environment—injecting synthetic failures (e.g., forcing the risk‑control service to timeout) and running the associated plan—validate that the plan still works. Successful rehearsals keep knowledge from being siloed; failures surface needed updates before a real incident.

Layer 5: Alert integration and graded automation

Integrating the plan platform with the monitoring/alerting system enables automatic plan recommendation when a specific alarm fires (e.g., “risk‑control timeout detected, suggest plan ‘Downstream Risk‑Control Failure’”). The operator then confirms parameters and executes. Automation can be graded:

Pure manual : Human decides and triggers each step.

Semi‑automatic : System executes but pauses for human approval at critical checkpoints.

Full automatic : System detects, decides, executes, validates, and rolls back without human intervention for low‑risk, high‑certainty scenarios.

The degree of automation must match risk; high‑impact actions retain human approval to avoid catastrophic mis‑fires.

Putting the evolution together

The progression moves from a static wiki page to a structured, idempotent, reversible workflow; then to a platform that offers one‑click execution, gray‑scale rollout, and rollback; next adds permission, approval, and audit; then regular chaos rehearsals to keep the plan fresh; and finally integrates with alerts and adopts risk‑based automation levels. Each layer solves the shortcomings of the previous one, but every layer introduces trade‑offs such as added system complexity, maintenance overhead, and the need for disciplined rehearsal resources.

In practice, teams should adopt the layers that match their scale and risk profile. For million‑QPS services, structuring a few critical plans and enabling occasional rehearsals may suffice. For ten‑million‑QPS services, full platformization, chaos rehearsals, and alert‑driven automation become essential to shrink MTTR from hours to minutes.