How to Run Effective Incident Response Drills for Resilient Systems

Disaster Role‑Playing and Incident Response Drills

To increase system resilience, teams should run regular disaster role‑playing exercises that replay real production incidents from start to finish. Google refers to this practice as the “Wheel of Misfortune”. By reenacting a past incident, the entire response workflow—including detection, escalation, mitigation, and post‑mortem—is exercised in a controlled environment.

Regular Disaster Recovery Testing (DiRT)

Google’s Disaster Recovery Testing (DiRT) program schedules recurring tests that initially target high‑risk failure scenarios. As teams address the discovered weaknesses, those tests become automated and low‑risk, effectively turning “risk” into a predictable, repeatable test case.

Refined Testing and Automation

Testing is evolving from pure technical checks (e.g., “Can we restore a completely corrupted database?”) to “process‑fix” challenges that validate human‑in‑the‑loop procedures such as approval workflows, on‑call hand‑offs, and notification routing. Technical checks can be scripted and run automatically, whereas process‑related checks often require manual verification and may expose hidden bottlenecks (e.g., a single approver who does not respond promptly).

Preparing Responders

Incident‑response drills help identify weak procedures, quantify their probability and impact, and build confidence for responders. Beyond technical competence, drills address psychological readiness: high‑stress incidents can cause fatigue, anxiety, and burnout, so managers should monitor responder well‑being and provide support when needed.

Writing Incident Response Tests

A practical way to start is to review recent incidents and answer three standard questions:

What went wrong?

What went well?

Where did luck play a role?

For each identified failure, create a small, focused test that verifies the fix and guards against regression. Example steps for a monitoring‑notification failure might be:

# Verify that an alert triggers a notification
curl -s http://monitoring.example.com/trigger?alert=test
# Check that the notification service receives the alert
curl -s http://notification.example.com/queue | grep test

Begin with simple, automated tests. As confidence grows, extend the suite to cover more complex, non‑technical aspects such as on‑call escalation paths, manual approval steps, and communication channels.

Maintain a steady cadence to avoid test fatigue. A common pattern is a one‑hour testing session every four weeks, which balances effort with continuous value. Over time, adjust frequency and depth based on observed risk reduction and team capacity.

For the original SRE discussion, see https://sre.google and the detailed chapter at https://martinliu.cn/blog/anatomy-of-an-incident-ch2/.