From Guesswork to Math: Modeling Capacity for Ten‑Million QPS Services

What capacity evaluation actually answers

Capacity evaluation, from a service‑architecture perspective, seeks to answer three critical questions: Will the service survive the target traffic? How many machines or CPU cores are needed? Will doubling the traffic break the system? These questions drive the entire capacity‑design system, influencing auto‑scaling, resource reservation, and mixed‑tenant scheduling.

Why experience‑based estimation fails at massive scale

Traditional estimation relies on historical peaks multiplied by a safety factor. While cheap and fast for low traffic, it becomes dangerously inaccurate at ten‑million QPS because a small percentage error translates to hundreds of machines and millions of dollars. Experience cannot reliably indicate capacity limits, bottlenecks, non‑linear fan‑out effects, version drift, or abnormal traffic spikes.

First pillar: Single‑machine benchmark to quantify per‑request cost

Isolate a service instance and stress it to its limits while recording CPU, memory, I/O, connections, and threads at each QPS level. For example, an order‑query service on a 4‑core, 8 GB machine shows a peak around 9 000 QPS, with CPU saturating and P99 latency jumping from 130 ms to 420 ms as load increases. This yields a “resource portrait” that breaks capacity into unit request cost × number of machines , exposing that linear extrapolation inside a single machine is already invalid.

Second pillar: Capacity model that bends the linear assumption

Use the Universal Scalability Law (USL) to capture contention and coherency penalties that cause non‑linear scaling. Fit USL to data from small‑scale clusters (1, 5, 20, 50 nodes) to locate the scalability knee N*. Below the knee, adding nodes is effective; near the knee, sharding or removing global coordination becomes necessary. Little’s Law (average concurrency = throughput × average response time) further translates target QPS and latency into required thread‑pool or connection‑pool sizes, highlighting that utilization near 100 % causes exponential queueing delay.

Third pillar: SLO‑constrained effective capacity

Theoretical peak capacity is not usable if it violates latency SLOs. In the benchmark table, the instance can reach 9 000 QPS, but P99 latency exceeds 50 ms beyond ~6 000 QPS. Therefore, the effective capacity is the maximum throughput that still satisfies the SLO, typically expressed as a “water‑mark” (e.g., CPU ≤ 60 %, P99 ≤ 50 ms).

Fourth pillar: Full‑link shadow testing

Individual service models ignore shared resources across a call chain. Shadow traffic—realistic request streams marked for isolation—runs through production‑like environments, routing reads/writes to shadow databases, tables, and caches. This reveals the true system‑wide bottleneck (the “shortest board” of the bucket) that single‑service tests miss, at the cost of building a shadow‑isolation framework and managing safety risks.

Keeping the model alive: Online water‑marks and continuous measurement

Models decay as code, traffic patterns, and business logic evolve. Continuous pipelines run baseline benchmarks after each release, while online dashboards monitor real‑time water‑marks (CPU, memory, QPS, latency, connections). Deviations between predicted and observed water‑marks trigger model recalibration. FinOps‑style cost per 10 k QPS translates technical capacity into business‑level budgeting.

Predictive capacity

Combine traffic forecasts (based on historical data, promotion calendars, growth trends) with the calibrated capacity model to compute required resources days in advance, enabling proactive auto‑scaling policies and pre‑reserved capacity.

No silver bullet: A continuously calibrated engineering system

Each step—experience, single‑machine benchmark, USL/queueing model, full‑link testing, online water‑marks, and predictive capacity—covers the blind spots of the previous one, but none alone solves the problem. The practical approach is to combine all four pillars into a self‑adjusting capacity model that evolves with each code change and traffic shift, turning “can it survive?” from a gamble into a verifiable engineering decision.