Capacity Prediction at 10M QPS: From Gut Feel to Data-Driven Decisions

Why Experience Fails at Scale

A typical scenario: business says "traffic will triple" for a Friday promotion. Platform asks whether that means users, orders, or QPS. The answer: "feels about the same as last time." Service owners then guess machine counts: gateway +50%, product service 2x, cache add one cluster, database wait and see. Actual results: homepage traffic only 1.8x, search 4.2x, orders as expected, but risk feature reads 6x due to rule upgrade. Most machines idle, a few dependencies saturated. The bottleneck wasn't CPU but shared storage write throughput.

The postmortem revealed the problem: "3x traffic" conflated users, business, requests, and resources into one number. Experience says leave headroom, but cannot answer which layer, for which time window, or how to adjust when wrong. At small scale, vague estimates are tolerable; at 10M QPS, a uniform 30% buffer means huge long-term cost while critical shards, connection pools, or cross-zone egress remain unprotected. "Overall headroom" and "critical-path headroom" diverge as systems grow.

Capacity prediction is not guessing a future QPS; it is using data to describe the range of future load and translating that range into executable, verifiable resource decisions.

Separate Demand, Load, and Capacity

Many capacity meetings jump straight to "how many machines." Machine count is a derived result, not the prediction target. A safer approach splits the problem into three layers:

Business demand : user or transaction behavior — DAU, concurrent users, searches, paid orders, upload volume. Closest to business plans, most affected by marketing, product changes, market shifts.

System load : work entering the technical system — ingress QPS, message write rate, DB query volume, cache read/write, log bytes. Conversion coefficients link demand to load: requests per active user, downstream calls per order, consumers per message.

Resource capacity : load a deployment can stably handle under latency, error-rate, and redundancy targets. Not lab peak throughput. The same 32-core instance yields different safe capacity under P99 < 100 ms vs P99 < 500 ms.

These layers cannot be permanently linked by fixed constants. A homepage change from one aggregated request to multiple parallel requests alters requests-per-user; cache hit-rate drop turns same ingress QPS into more DB queries; data growth raises per-query cost. Conversion coefficients must be monitored as time series.

The derivation simplifies to:

Future Resource Need = Future Business Volume × Load Amplification Factor ÷ Unit Resource Safe Capacity

All real work hides in the three variables: future volume has a prediction interval, amplification factor changes with versions and traffic mix, unit capacity is constrained by SLOs and failure scenarios. Data-driven prediction does not eliminate uncertainty — it makes uncertainty explicit in the model.

Predict business behavior first, derive system load, then compute resources. This is more explainable and makes it easier to trace where changes originate than directly extrapolating machine counts.

Data Foundation Determines Prediction Ceiling

Teams often obsess over model choice. In practice, the most common issue is unstable input metric definitions.

Same "request count": gateway counts received requests, service counts requests entering business logic, client may count retries. A release changes tracking tags and the curve drops; scaling causes instance churn and double-counting in per-instance sums; monitoring sampling loses the exact peak during incidents. Modeling dirty data produces confident garbage.

A usable data foundation for capacity prediction must do five things:

Unified Definitions and Dimensions

Each core metric needs explicit name, unit, aggregation method, sampling period, label dimensions, and owner. Example: ingress request count = success + failure, summed per minute, retaining region, business line, caller dimensions. CPU uses core-seconds, not percentage, to avoid mixing instance types.

Dimensions are not "more is better." High-cardinality labels increase storage and fragment prediction objects. Capacity layer keeps only dimensions affecting resource isolation and scheduling: region, AZ, tenant tier, service version, resource pool. Request-ID diagnostic dimensions are unsuitable for long-term capacity series.

Label Anomalies Instead of Deleting

Incidents, load tests, crawlers, backfills, monitoring gaps create outliers. Deleting hides real pressure; keeping all distorts the baseline. Better: tag anomalies — real business spike, failure-induced retry, collection error. Real spikes stay as future risk reference; failure retries isolated from normal prediction but fed into failure-capacity scenarios; collection errors fixed or imputed. Same spike, different cause, different model meaning.

Preserve Change Events

Time series show that a curve changed, not why . Releases, config changes, marketing, pricing, holidays, rate-limiting, DC migrations must enter an event calendar. Predictions can then separate cyclical from one-off changes and find references for similar future events.

Choose Granularity Matching Decisions

Quarterly procurement cares about weekly/daily trends; on-call scaling cares about minute-level peaks; DB sharding cares about data growth and migration windows. No single sampling granularity covers all decisions. Raw metrics stored at fine granularity; prediction layer builds multiple time windows aligned with resource lead times.

Continuous Data Quality Checks

Missing rate, latency, sudden shifts, label drift, cross-source discrepancies need quality alerts. A successful prediction run ≠ trustworthy prediction. If gateway and service request counts diverge 5% long-term then suddenly 30%, pause automated resource decisions and verify whether business paths changed or data broke.

With a solid data foundation, even simple models beat "pick a growth rate" because inputs, exceptions, and final errors are traceable, giving the team a shared factual basis.

Models Need Not Be Complex

Capacity curves typically contain baseline, trend, cycles, and event shocks. Weekday lunch peaks = daily cycle; weekend drop = weekly cycle; user growth = long-term trend; promotions and hot events = extra shocks. A practical framework decomposes them:

Predicted Load = Baseline + Long-term Trend + Cyclical Component + Known Event Deltas + Uncertainty Margin

Baseline from recent same-window median or weighted average. Trend from recent weeks' growth rate, capped against outliers. Cyclical component from historical same-weekday, same-hour distribution. Known event deltas from business plans and analogous past events. Margin from historical prediction error, resource lead time, and risk level.

Decomposition makes numbers auditable. A 40% week-over-week increase breaks down as 10% organic growth, 20% planned event, 10% safety margin. If the event cancels, remove its delta — no model retraining wait.

Complex models have value but shouldn't be the starting point. A two-week-old service lacks history for reliable seasonality. Use similar services, load-test results, and business plans for initial intervals with low confidence; replace priors as real data accumulates. For stable core services, simple same-period quantiles often work well: take last eight weeks' same-weekday, same-5-minute-window load distribution, adjust with recent trend, naturally preserving cycles and variance. If events drive load, add exogenous variables: event type, exposure slots, user reach.

Model choice must serve decision cost. If adding 10 stateless compute nodes is cheap, accept wide safety margin. If procuring specialized hardware or sharding a DB takes months, need longer horizons, stricter data governance, and multiple scenarios. Spend prediction precision on hard-to-reverse, long-lead-time resources.

Capacity prediction seeks not a smooth-looking curve, but a sufficiently early, sufficiently credible, explainable load range before resource decisions are made.

Express the Future as Intervals, Not Point Estimates

Future load is inherently uncertain. Reporting only "next week peak 3.2M QPS" implies 3.2M is a hard boundary. Better output includes median, upper bound, lower bound, confidence levels, and which risks are incorporated.

For elastic resources, run near median baseline and scale dynamically via short-term forecasts. For slow-to-expand, high-failure-cost resources, use higher-quantile upper bounds. Resource decisions fundamentally trade stockout cost against idle cost.

Example: a service's forecast distribution shows normal peaks 2.8M–3.2M QPS, rare event scenarios up to 3.6M QPS. If nodes scale in 3 minutes with reliable upstream queuing and rate-limiting, static capacity need not cover 3.6M. If DB scaling takes weeks and saturation causes hour-long recovery, plan early for the high scenario.

Interval width also reflects data quality. New business with no history → wide interval; mature stable cycles → narrow interval. An unusually precise number for a new business signals dishonesty about unknowns, not model prowess.

Teams should maintain multiple scenarios, not just "normal" and "worst case." Common practice: baseline, growth, stress. Baseline uses known plans; growth adds higher conversion or accelerated organic growth; stress adds local failures, cache misses, hotspot skew. Each scenario must bind actions and trigger conditions, avoiding decorative curves in slide decks.

Translate Predicted Load into Resources

Forecasting QPS completes only half of capacity planning. Next, convert load into instances, shards, bandwidth, quotas, and verify they meet SLOs under failure scenarios.

Step 1: obtain unit resource safe capacity. Load tests give a starting point but cannot directly use peak throughput as production capacity. Safe capacity must simultaneously satisfy CPU, memory, latency, error rate, queue length, downstream pressure constraints, and reflect production request mix. If test requests are all light but production includes complex queries and large payloads, test results are optimistically biased.

Step 2: identify saturation signals. CPU saturation is only one. A service may queue at 55% CPU due to thread-pool and downstream connection-pool exhaustion; cache nodes may hit network bandwidth limits; DBs may hit single-shard hotspots and log-write limits. Unit capacity is determined by the resource that hits its safety boundary first.

Step 3: factor in failure redundancy. 100 instances each carrying 1% traffic does not guarantee safety at <100% average utilization. If design requires surviving one AZ loss, remaining AZs must absorb traffic without breaching safe capacity. Rolling releases, instance restarts, data rebalancing also temporarily reduce available capacity.

Simplified instance check:

Instance Count = Predicted High Load ÷ Per-Instance Safe Throughput ÷ Target Availability Ratio

Target availability ratio encodes failure and maintenance headroom. If the pool must survive any single AZ loss, not all nodes' theoretical capacity counts as daily consumable capacity. The ratio should be computed from deployment topology, recovery time, and service tier — not a company-wide fixed 70%.

Also check dependency amplification. One ingress request may trigger 4 cache reads, 2 DB queries, 8 messages. Scaling only ingress pushes pressure downstream. Distributed tracing and service topology provide call multipliers; business change reviews must declare whether multipliers will change.

Capacity is not a resource's theoretical ceiling; it is the safe operating envelope under target latency, target error rate, and defined failure models.

Turn Errors into a Closed-Loop Signal

Predictions will be wrong. Mature systems don't try to prove zero error; they continuously answer: how much, why, does this error change the next decision?

Baseline practice: version every forecast instead of overwriting the same curve daily. Then compute error at each lead time — 15 minutes, 1 day, 2 weeks, 1 quarter. Short-term forecasts serve elasticity; long-term forecasts serve procurement and architecture changes; they cannot share a single error metric.

Error metrics need business context. MAPE is intuitive but distorts on low-volume series; MSE penalizes large deviations more; quantile forecasts check coverage proportion. For capacity, under- and over-prediction costs are usually asymmetric — track them separately.

Forecast errors must be viewed alongside capacity outcomes. Over-predict 20% but resources release in minutes → small cost impact. Under-predict 5% causing a missed monthly DB sharding window → large consequence. Model evaluation weights should align with resource cost and lead time.

An often-ignored loop signal: did the resource action actually deliver expected capacity? Requesting 20 instances ≠ 20 effective instances immediately. Image pull, data warm-up, LB convergence, cache refill all take time. The prediction system should record the full cycle from request → delivery → capacity effectiveness to calibrate actual lead time.

When actuals approach the predicted upper bound, tiered actions trigger:

Verify data quality and business events — confirm not a metric anomaly.

Scale elastic resources, monitor unit capacity for shifts.

For slow resources, start manual approval, quota coordination, migration planning.

If resources cannot arrive in time, preemptively enable rate-limiting, degradation, traffic shifting.

Post-event, reclaim resources and feed actual peak and action effectiveness back into the model.

Start Building with a Single Spreadsheet

Data-driven ≠ build a massive prediction platform day one. A realistic evolution: pick one service with slow resource delivery, high business value, relatively complete history, and run the minimum viable loop.

Phase 1 — Explainable Baseline : Unify metric definitions, save 8–12 weeks of same-period history, add known event plans, output baseline, upper bound, key assumptions. Even in a spreadsheet, weekly versioned updates already beat verbal estimates.

Phase 2 — Load-to-Resource Mapping : Via load tests and production observation, determine per-instance safe capacity, fill in failure redundancy and delivery lead time. Every number in the resource plan must trace back to business volume.

Phase 3 — Automation : Short-term forecasts drive stateless service scaling; long-term forecasts create quota and procurement tasks. Automate low-risk resources first, with max adjustment limits, cooldowns, and human takeover conditions.

Phase 4 — Hierarchical Prediction : Align site-level, region-level, business-line, service-level, and resource-pool forecasts, propagate load through dependency topology. Platform focus shifts from algorithms to metric governance, model versioning, permissions, audit, and fallback.

Beware two extremes: (1) senior engineers still dictate conclusions, just filling numbers into the system — not truly data-driven; (2) blind faith in the model, ignoring imminent business changes absent from history — treating unknown as non-existent. In practice, humans and models divide labor: models handle stable patterns, massive series, error statistics; business and engineering add upcoming events, architecture changes, risk appetite. All manual adjustments must record reason and expiry, auto-expiring after the event to prevent one-time boosts from permanently polluting future forecasts.

Make Every Prediction Interrogatable

Moving capacity prediction from experience to data does not replace the owner's judgment with an algorithm API. The real change: every resource number can answer which business assumptions it came from, which historical window was used, which failure scenarios are covered, what error margin is allowed, and when it must be recalculated.

At 100K QPS, a few system-savvy people can watch key resources. At 1M QPS, stable metric definitions, load-test baselines, and periodic capacity reviews are needed. At 10M QPS, business plans, dependency amplification, prediction intervals, resource lead times, and error feedback must form a closed loop. The larger the system, the less a uniform buffer can hide local uncertainties.

Experience remains vital, but it must not stay at "I think it's enough." Write experience as event hypotheses, turn history into comparable baselines, turn unknowns into prediction intervals, turn resource actions into verifiable outcomes. Only then can the team choose before traffic arrives, instead of proving whose hunch was right after alerts fire.

Good capacity prediction never promises the future will be a single number. It tells you in advance which futures are most likely, and what to do when each arrives.

Next time you hear "traffic will roughly triple," ask: which call chains will that 3x propagate through, what resource will exhaust first, and how much decision time does the system actually have left?