How to Ensure System Stability During Massive Sales Events: A Complete SRE Playbook

Introduction

Every year large‑scale promotions demand stable systems; common practices include full‑link load testing, capacity assessment, throttling, and emergency plans, but the underlying reasons and theoretical foundations are often overlooked.

What Defines a Stable System?

Google SRE’s Dickerson Hierarchy of Service Reliability describes a pyramid where the base is Monitoring , followed by Incident Response , Postmortem & Root‑Cause Analysis , then Testing & Release Procedures , Capacity Planning , and finally Product Design and Software Development at the top.

Big‑Promotion Stability Assurance Methods

The goal is to systematically protect services during short, high‑traffic events (typically ~2 months). The approach focuses on identifying critical links, analyzing traffic data, and strengthening monitoring, capacity, incident response, testing, and post‑mortem processes.

System & Biz Profiling

Monitoring

Capacity Planning

Incident Response

Testing

Postmortem

1. System & Biz Profiling

Map the entire system from entry points (HTTP, RPC, messaging) to downstream services, classifying nodes by dependency strength, availability, and risk. Produce data on core links, strong/weak dependencies, and financial‑loss exposure.

Entry Point Inventory

Core high‑SLI traffic

Revenue‑critical flows

High‑volume traffic (top TPS/QPS)

Node Layering

Strong vs. weak dependencies

Low‑availability nodes

High‑risk nodes (recent upgrades, no prior load tests, etc.)

2. Monitoring

Adopt white‑box monitoring (business, application, system layers) and ensure coverage for all critical links. Use the four golden metrics – Latency, Error, Traffic, Saturation – to define alerts, setting appropriate thresholds and notification channels.

3. Capacity Planning

Balance risk minimization and cost efficiency by estimating peak traffic, converting it to resource capacity using Little’s Law and N+X redundancy principles. Distinguish regular traffic from irregular spikes caused by marketing or disaster‑recovery scenarios.

These calculations provide only an initial estimate; final capacity must be validated with periodic stress testing.

4. Incident Response

Develop pre‑emptive and emergency response plans categorized as technical or business, pre‑emptive or emergency. Define execution and closure criteria, trigger thresholds, impact scope, responsible personnel, and verification steps.

5. Operational Playbook

Structure the playbook into pre‑event, during‑event, and post‑event phases, including checklists, pre‑emptive plans, emergency scripts, alert dashboards, upstream/downstream machine groups, on‑call duties, core broadcast metrics, contact lists, and incident records.

6. Post‑Event Review

After the promotion, execute recovery tasks (throttling adjustments, scaling down) and conduct a thorough post‑mortem to capture lessons learned.

7. Drill Exercises

Run tabletop or sandbox drills using real historical incidents to validate response procedures, focusing on rapid service restoration, targeted diagnosis via white‑box metrics, coordinated hand‑offs, and escalation decisions.