10M QPS Architecture #256: Shifting Mega-Promo Reliability from Reactive to Proactive

At 7:58 PM, two minutes before a major promotion launch, the war room reports all systems green. At 8:00 PM traffic spikes; within three minutes cache hit rates drop, database connection pools queue, and retries amplify a local slowdown into a full-chain outage. Hundreds of alerts fire simultaneously, and the team struggles to decide which action to take first. Scaling instances waits for image pulls; raising timeouts consumes more connections; rolling back config may affect active users. The scarce resource is not machines but the ability to make correct judgments in minutes.

Teams often mistake mega-promo readiness for simply adding machines and staff. Post-mortems reveal machines were not exhausted and staff were plentiful. The flaw lies in preparation: capacity estimated from average traffic, stress tests hitting only the entry point, runbooks saying "scale on anomaly," alerts describing resource symptoms, and critical decisions relying on ad-hoc improvisation.

The dividing line of mega-promo readiness is whether unknown risks have been converted into verified capacity, switches, runbooks, and decision criteria before traffic arrives. Proactive readiness does not promise zero failures; it ensures that when failures occur, the system has boundaries, the scene has signals, actions have sequence, and the business knows what to sacrifice. Reactive mode relies on heroics; proactive mode relies on a repeatable engineering system.

Why Stable Systems Still Lose Control During Mega Promos

Daily traffic resembles a gentle slope; promo traffic is a step function. The peak difference is superficial; the real challenge is simultaneous changes in traffic shape, user behavior, and system state.

Before launch, users linger on the venue page; at launch they simultaneously refresh, claim coupons, check inventory, and place orders. Requests no longer follow smooth arrival assumptions, and hot items concentrate data access. A system that ramps to peak over 10 minutes may not survive the same ramp in 10 seconds. Auto-scaling depends on metric windows, instance startup, and service warm-up, often missing the first spike.

Promo chains are longer than daily chains. Normal browsing may traverse gateway, product, and cache; promo requests add marketing, coupons, inventory, risk control, order, payment, and messaging. Each synchronous dependency reduces overall success rate and increases tail latency. Ten serial services each at 99.9% availability yield a theoretical success rate below 99%.

Another overlooked change: promos rewrite system parameters. New product pools, coupon rules, inventory strategies, risk thresholds, and degradation switches deploy en masse before the event. The system faces not just higher traffic but a changed business state. Many incidents blamed on capacity actually stem from config errors, unwarmed caches, hot shards, or downstream quotas not adjusted.

Mega promo is not daily traffic multiplied by a factor; it puts traffic, dependencies, data, and organizational collaboration into the same high-pressure window.

This explains why single-machine CPU or cluster average utilization often misleads. Averages hide hotspots; resource metrics lag business failures; entry throughput cannot reveal internal amplification. You must first understand how promo traffic traverses the system before discussing resource preparation.

Deriving Guardrails from Business Objectives

Readiness work often starts with "expected QPS." That question matters but should not be first. The first question: which business outcomes must be protected during the event?

For e-commerce, core outcomes may be: users can enter the venue, see sellable inventory, orders are not duplicated, and funds are correct. Simplified recommendations, temporarily invisible comments, or delayed logistics updates are usually acceptable. Without tiering business criticality, every service labels itself core, making resource allocation and failure trade-offs impossible.

Goals can be split into four tiers:

Non-negotiable correctness: Funds, inventory, order idempotency, and user rights must not break under pressure. Better to reject some requests than return wrong results.

Key availability for the promo: Login, product detail, checkout, and payment must meet explicit success-rate and latency targets. Use business metrics (order success rate, payment success rate, sellable rate) not just service availability.

Degradable experience: Personalized recommendations, complex sorting, real-time comments, and non-critical displays can switch to simplified versions under pressure. Degraded pages must remain understandable, not return generic errors.

Deferrable background tasks: Reporting, non-real-time sync, profile updates, and some message consumption can pause or shed load to free resources for the online chain.

After tiering, attach numbers. Example: within the peak 5 minutes, checkout success rate must not fall below a business red line; inventory oversell must be zero; core page P99 latency must stay under target; message backlog must catch up within 30 minutes post-event. Thresholds are jointly agreed by business and tech, not copied.

A practical test: a goal must directly trigger action. "Good user experience" cannot decide degradation; "payment success rate below 99.9% for 2 minutes → freeze non-essential changes and switch to backup channel" can guide the scene.

Resource budgets, degradation order, and alert thresholds must derive from business red lines, not from each service guessing independently.

Capacity Estimation Must Cover the Full Chain

A common promo capacity formula: historical peak × growth factor × safety factor. It serves as a first draft, not the final answer. The same entry QPS, after cache misses, retries, fan-out, and async consumption, becomes vastly different internal load.

A useful capacity model includes at least six variables:

Entry peak and ramp slope, including second-level spikes, not just minute averages.

Request mix per interface (browse, coupon claim, order, query proportions).

Call amplification factors: synchronous fan-out, retries, internal polling.

Data access distribution: hot items, hot shards, large-customer concentration.

Service steady-state vs. burst capacity and sustainable duration.

Dependency quotas: DB connections, cache bandwidth, message partitions, third-party APIs, network egress.

Example: entry 1M QPS, 10% enter trade pre-check; each check calls 3 downstreams; client and gateway retry once on error. Surface shows 100K core requests; internally it may amplify to 600K calls. If a downstream slows, unfinished requests hold threads and connections, further reducing usable throughput.

Capacity tables should follow the user journey. Each node records entry proportion, amplification factor, single-instance safe throughput, target instance count, key resources, and bottleneck evidence. "Safe throughput" is not the maximum from stressing a machine to its limit, but the sustainable upper bound where target latency, error rate, GC, and queue length all meet requirements.

Safety factors should not be a uniform 2× across the chain. Nodes with slow scaling, high failure impact, or little historical data need larger margins; nodes with mature elasticity, fast degradation, and ample stress-test evidence can be tighter. Safety factor prices uncertainty: less evidence → larger margin.

At ten-million QPS scale, resource scale itself changes risk. Multi-cluster, multi-region, multi-tenant deployments mean any average can mask local saturation. Capacity management units must descend from "total service instances" to region, cluster, machine type, shard, and hotspot group. You must know which piece will saturate first, not just total headroom.

Capacity estimation output should not be a machine procurement list, but a chain map that pinpoints the first bottleneck and failure mode.

Stress Testing Aims to Expose System Boundaries

Some teams treat stress testing as an exam: hit target QPS, screenshot, done. That yields a nice transcript; the real boundaries remain unknown.

An effective stress test must answer four question categories:

Steady-state ceiling: Gradually increase load, observe the sequence where throughput stops scaling linearly, P99 latency rises, error rate climbs, queues back up, and resources saturate. Do not crush the system immediately; otherwise you only learn it breaks, not where instability begins.

Burst absorption: Simulate the promo's step and pulse traffic; verify rate limiting, connection establishment, cache warm-up, and auto-scaling. Steady 1M QPS does not guarantee jumping from 200K to 1M in 10 seconds.

Failure convergence: Inject cache node failure, DB read-only switch, message partition anomaly, network jitter, downstream latency. Observe if retries are controlled, circuit breakers trip timely, degradation works, and how backlog digests after recovery.

Post-event recovery: Promo does not end at peak. Message backlog, order compensation, reconciliation, and log writes create a second wave. System must catch up background tasks without affecting online traffic.

Environment choice depends on risk. Fully isolated envs are safe but hard to replicate production data distribution, network topology, and dependency quotas. Full production stress is most real but highest risk. Common layered approach:

Single-service stress to get component boundaries and initial capacity model.

Shadow env to verify full chain and data construction.

Production small-flow shadow to validate real topology and observability.

Production capacity verification under explicit isolation, rate limits, and stop conditions.

Each level needs auto-stop conditions: core success rate below red line, DB replication lag exceeding threshold, payment chain showing unacceptable errors. Stopping stress test is not failure; it proves protection mechanisms took over before risk spread.

Stress test data must avoid self-deception. Traffic models from real user paths and historical events, not uniform random requests; datasets must create hotspots and large objects; clients must measure send vs. complete rates to avoid generator bottlenecks; downstream mocks only for isolation tests, not full-chain validation.

After stress testing, the team should have three key deliverables:

Verified safe capacity and first bottleneck.

Metric signatures from normal to instability.

System behavior and recovery time after protection actions engage.

The most valuable stress-test artifact is not the peak number, but evidence of "when to take what action."

Turning Degradation, Rate Limiting, and Isolation into Actionable Capabilities

When the system hits its boundary, the scene has three levers: reduce incoming work, lower per-request cost, isolate the abnormal part. In engineering terms: rate limiting, degradation, isolation.

Rate limiting controls entry volume. It must prioritize core user journeys and differentiate by tenant, region, interface, and business tier. A single global threshold lets low-value queries crowd out transactions and lets a hot tenant drag others. Rate-limit responses must be explicit: queueable requests enter bounded queues; non-queueable requests fail fast to prevent client retry storms.

Degradation reduces per-request cost. Disable complex recommendations, use static venue pages, skip non-core checks, read slightly stale snapshots — all concentrate resources on the critical path. Degradation design must pre-confirm correctness boundaries. Steps involving funds, inventory, permissions cannot be skipped just because they are slow; they need stricter rate limiting or queuing.

Isolation contains failure blast radius. Thread pools, connection pools, queues, cache shards, and clusters can serve as isolation boundaries. If marketing and checkout share a connection pool, marketing jitter can exhaust checkout resources. If promo and daily traffic fully co-run, promo anomalies may affect regular users. Even with ample capacity, logical or physical isolation should be retained.

Protection capabilities need four attributes:

Pre-provisioned: Rules, switches, permissions exist before the event; no live code pushes or ad-hoc approvals.

Gradual rollout: Enable on a small traffic slice or single region first, confirm effect, then expand — avoiding the mitigation itself becoming a new incident.

Observable: After a switch flips, the system shows which requests were degraded, how much resource was saved, and whether core metrics recovered. "Operation succeeded" is far from enough.

Recoverable: As pressure eases, exit protection in order to prevent all background tasks and non-core functions from restarting simultaneously, causing a second shock.

The scene fears a single "degrade" master switch. No one knows exactly what it turns off or if data stays safe, so nobody dares press it. Good runbooks break actions into small pieces, documenting impact, owner, verification metrics, and rollback.

Observability Designed Around Decisions

Promo dashboards often cram CPU, memory, disk, network, and dozens of service curves. The screen looks busy, yet when anomalies hit it cannot answer "Can users still place orders?" and "What do we do next?"

Proactive readiness needs an observation chain from business outcome to resource root cause.

Top layer — Business outcomes: Venue reachability, coupon claim rate, order success rate, payment success rate, inventory accuracy, GMV progress. Judges user impact and business red lines.

Second layer — User journey: Throughput, errors, latency for key steps (page, checkout, order, payment). Locates which step loses business.

Third layer — Services & dependencies: Golden signals for gateway, app, cache, DB, messaging, external APIs, plus retry, circuit breaker, degradation, queue states.

Fourth layer — Resources: CPU, memory, connections, bandwidth, file handles, thread pools. Explains upper-layer changes; cannot replace upper-layer goals.

Change events must overlay the same timeline. Releases, config changes, scaling, traffic shifts, degradation, fault injection — all tagged with operator, scope, version. When success rate drops at 20:03, the scene sees that at 20:02 a region switched marketing rules, instead of digging through chat logs.

Alerts should also be organized by decision tier:

Early warning: Risk approaching boundary (connection pool usage rising, cache hit rate trending down). Triggers investigation and preparation.

Action alert: Execute predefined runbook (order success rate crosses red line → trigger non-core degradation).

Incident alert: User impact occurred; enter incident command process.

If one root cause spawns 300 instance alerts, converge them into one event with affected scope. The scene needs executable context, not message count.

A good alert answers at least four things: who is impacted, which target is deviated, what change might be related, where to look or what to do next.

Also pre-check observability system capacity. Promos increase metric labels, log volume, trace volume. If observability collapses first, the scene goes blind when evidence is most needed. Sampling strategies, log rate limiting, dedicated channels for core metrics, and dashboard query capacity must be part of capacity planning.

Runbooks: From Documents to State Machines

Many runbooks are long but the actionable part is one line: "scale or degrade if necessary." They record what might be done, not when, by whom, or what to verify after.

An executable runbook resembles a state machine. Each state has entry conditions, allowed actions, owner, verification metrics, and exit conditions.

Example promo states:

Normal: Core metrics meet targets; protection switches on standby.

Watch: Leading indicators deteriorate; freeze non-essential changes; prepare degradation.

Under Pressure: Business metrics near red line; activate non-core degradation and local rate limiting.

Incident: Core business crosses red line; incident commander takes unified command.

Recovery: Metrics stabilize; exit protection in batches and process backlog.

Each action card stays concise:

Runbook scaling actions must state time-to-effect. If new instances need 15 minutes from request to warm, they only address sustained pressure, not the first 30-second spike. For bursts, pre-scaling, cache warm-up, rate limiting, and fast degradation are more reliable. Writing an action that cannot take effect in the first 5 minutes into the runbook creates false confidence.

Scene operations must have a single entry point and audit trail. Direct machine login, direct DB writes, or bypassing the platform for traffic shifts may work short-term but detach real state from the control plane. Emergency permissions can simplify approval but cannot remove scope confirmation, two-person review, and operation logging.

War Room Needs Clear Roles, Not More People

Dragging all dev, QA, product, and ops into one group before the event does not automatically speed handling. When an anomaly hits, dozens of people asking, posting screenshots, and executing ops simultaneously drowns critical information.

A lean incident collaboration structure typically includes:

Incident Commander: Judges severity, sets current objective, approves high-risk actions, coordinates resources. Commander should not deep-dive a single machine; otherwise no one maintains global view.

Technical Lead: Aggregates service investigations, forms hypotheses, proposes actions. Domain on-call provide evidence, execute authorized ops.

Business Lead: Assesses user impact and business trade-offs (e.g., shut which marketing capability, extend promo, external communication).

Scribe: Maintains unified timeline recording phenomena, hypotheses, decisions, actions, results. Serves both live handover and post-mortem.

Communications Lead: Syncs to external teams and management, preventing everyone from interrupting troubleshooters.

Collaboration channels can be split: one for command & decisions, one for technical investigation, one for business sync. Key conclusions written back to the unified timeline. Allows deep discussion without drowning command signals.

The scene should run on a fixed cadence, e.g., every 5 minutes update: current user impact, known facts, hypotheses under test, actions taken & effect, next decision point. Frequency adjusts with severity; template predefined.

The most important organizational principle at an incident scene: at any moment there is one clear objective, one decision maker, and one credible timeline.

This does not weaken expert judgment. It lets experts focus on evidence and options without also carrying cross-team coordination. The commander is not the highest rank but the trained person who can maintain decision process under pressure.

Drills Verify People, Systems, and Runbooks Work Together

If a protection switch has never been activated in the real chain, the runbook is an uncompiled program. The page may not open, permissions may have expired, switch semantics may have changed, on-call may not know where to observe results.

Pre-promo drills must cover three layers:

System drills: Verify technical mechanisms — rate limiting, degradation, traffic shifting, scaling, failover, cache rebuild, backlog recovery.

Process drills: Verify collaboration — participants discover, escalate, command, record, decide without knowing the fault script; observe if information reaches the right roles.

Business drills: Verify trade-offs — simulate core metric drop, require business lead to choose which promo capabilities to cut, confirm degraded pages, user prompts, compensation strategies are acceptable.

Drills need fault injection but not theatrical effects. Pick the most likely high-impact failures: hot cache node loss, regional DB latency, third-party payment quota exhaustion. Each drill validates a few explicit hypotheses; afterward update capacity, runbooks, tools.

Drill quality measured by time breakdown:

Time from anomaly appearance to detection.

Time from detection to confirmed user impact.

Time from confirmation to mitigation decision.

Time from action execution to core metric recovery.

Time from recovery to backlog and protection state cleanup.

These times drive improvement more than "drill passed." If detection takes 1 minute but authorizing a degradation takes 12 minutes, next step is optimizing permissions and decision flow, not adding another monitor.

Drill findings must enter the promo readiness checklist with owner and deadline. Unfixed issues turn drills into theater: fault found, meeting held, next time stuck at same spot.

Readiness Gates Close the Preparation Loop

Near promo, every team says "basically ready." Proactive readiness demands a verifiable definition of done. Go/no-go decided by verifiable gates.

Gates organized across dimensions:

Business goals confirmed: core journeys, correctness red lines, degradable scope agreed.

Capacity model complete: entry, internal amplification, hotspots, external quotas evidenced.

Target stress & fault scenarios tested; first bottleneck and stop conditions clear.

Key switches drilled; permissions, owners, verification metrics, rollback ready.

Promo configs and releases pass pre-freeze check; change windows and exception process clear.

Observability drills from business outcome down; observability system capacity sufficient.

On-call roster, command roles, contacts, escalation paths tabletop-tested.

Residual risks signed off by business with accepted scope and compensation plan.

Gates are not to block launch on the last day. Effective gates set early checkpoints: 4 weeks before — initial capacity assessment; 2 weeks — full-chain stress test; 1 week — drills; 2 days — only whitelisted changes. Later discovery means fewer fix options, higher risk cost.

Readiness meeting allows three outcomes: Pass, Pass with Risk, Fail. Pass with Risk must document risk owner, trigger condition, compensation action. If everything ends up "Pass," gates quickly lose credibility.

On promo day, readiness needs continuous validation. Traffic forecasts update, third-party quotas change, a cluster may lose capacity from another incident. Final pre-launch check is not reading the checklist aloud but confirming key assumptions still hold.

After the Event, Readiness Is Only Half Done

When traffic drops, war rooms often disband quickly. Yet the system may still be degraded, messages backlogged, temporary scaling not reclaimed, alerts silenced, manual permissions open. Next-day data gaps or cost anomalies often originate from this ignored recovery window.

Recovery must be sequenced:

Confirm core business stable and observe long enough to cover metric windows.

Exit rate limiting and degradation in small batches, each time verifying resource levels and business metrics.

Restore background tasks by priority, avoiding all consumers chasing backlog simultaneously.

Revoke temporary routing, whitelists, scaling, alert suppressions one by one.

Complete order compensation, fund reconciliation, inventory verification.

Post-mortem should not only tally incidents. Even with zero user-visible faults, compare predictions vs. reality:

Predicted peak vs. actual peak delta.

Which node came closest to capacity limit.

Which degradations or rate limits actually triggered.

How many alerts guided action.

Which decision waited longest.

Which runbooks were unused — because no risk or because not executable.

Whether temporary resources and manual effort were justified.

These data update the next capacity model and readiness standard. Persistent prediction bias → adjust traffic model. Auto-scaling always misses spikes → add pre-scaling or shorten warm-up. A degradation repeatedly proves ineffective → redesign protection boundary.

A simple maturity path shows where the team stands:

Proactive readiness is not a one-off project before a promo; it embeds capacity validation, fault drills, and decision training into daily engineering rhythm.

As systems scale from millions to tens of millions QPS, staring at every chain manually becomes unrealistic. Larger scale means more frequent local failures, greater cross-region variance, and harder-to-estimate blast radius of ad-hoc ops. Mature practice does not chase zero errors everywhere; it isolates errors, prioritizes critical journeys, and triggers protection actions on evidence.

Before the next promo, ask the team a concrete question: If at the 90th second after peak, order success rate starts dropping, does the scene know who, by which minute, based on which metric, will trigger which switch, and can confirm it actually worked?

If the answer is still "we'll discuss in the war room," mega-promo readiness remains in reactive phase.