From Zero to Ten‑Million QPS: How to Reserve Resources for Guaranteed Capacity

On a typical Wednesday afternoon a mistakenly‑run full‑table scan batch job consumed all CPU and memory bandwidth on a shared host, causing the payment service’s P99 latency to jump from 80 ms to 2 s. This classic *noisy‑neighbor* scenario illustrates the danger of a pure best‑effort shared pool where critical paths have no capacity guarantee.

Best‑effort shared pool and its four hidden risks

When a system is small, resources are allocated on a best‑effort basis: services compete for a common pool without any reservation. This model works while traffic is low, but it embeds four risks that explode at scale:

Noisy neighbor : an unrelated burst (e.g., a batch report) can starve critical services.

No capacity guarantee : core services rely on luck to obtain resources during traffic spikes.

Reactive scaling : capacity is only added after alarms fire, leading to cold‑start delays.

Uncontrolled blast radius : in multi‑tenant platforms a single tenant’s overload can affect all others.

At ten‑million‑QPS, these risks become unacceptable because any contention can cascade across the call chain and cause a full‑stack avalanche.

Why reservation becomes mandatory at ten‑million QPS

At million‑QPS levels occasional contention is tolerable, but once traffic reaches ten‑million QPS the system has longer call chains, more dependencies, sharper traffic peaks, and larger variance. A single resource contention can instantly trigger a chain reaction, so critical services must have deterministic capacity rather than probabilistic competition.

Planned traffic surges (e.g., sales events) further demand pre‑locked capacity so that services never have to scramble for resources when the load spikes ten‑fold.

Five practical reservation mechanisms

1. Resource quotas (requests / limits)

The most basic reservation is assigning each service a clear quota. In Kubernetes this is expressed by the requests and limits pair: requests tells the scheduler the minimum amount of CPU or memory a container needs and reserves it on the node; it is the true guarantee for a critical service. limits sets an upper bound; exceeding it triggers throttling (CPU) or OOM kill (memory). Limits act as a guardrail to prevent a service from becoming a noisy neighbor.

At a larger granularity, ResourceQuota caps total resources for a namespace (team or business line) and LimitRange provides default and min/max values for each pod, ensuring no single line can consume the whole pool.

Similar concepts exist in thread pools ( maxPoolSize), semaphores, and database connection pools ( min-idle, max-active), where min-idle is a form of reservation that keeps a ready‑to‑use pool of connections.

2. Reserved instances

Beyond quotas, services can lock capacity ahead of time by provisioning reserved instances. For a major sales event, you might expand a critical service from 50 to 200 pods a day in advance and keep the extra 150 pods warm and ready.

Cloud providers offer this as Reserved Instances (AWS) or Reserved Instance Coupons (Alibaba Cloud), trading a multi‑year commitment for lower price and guaranteed capacity. This reduces cost by 30‑70 % for steady‑state core services while ensuring machines are available when needed.

Capacity pools or hot‑standby instances work similarly: a set of pre‑started instances sit idle until the primary pool is exhausted, then take over within seconds.

3. Resource isolation

Isolation separates critical and non‑critical workloads at the physical or OS level. cgroups and namespaces provide hard CPU/memory isolation for containers. Deploying critical services on dedicated machines or racks eliminates cross‑contamination.

Within a process, thread‑pool isolation (e.g., Hystrix, Sentinel) ensures a slow downstream service only blocks its own pool, not the entire service. Connection‑pool isolation applies the same principle to databases.

4. Priority and preemption

When resources become scarce, Kubernetes PriorityClass and preemption let high‑priority pods evict low‑priority ones. By labeling payment pods as high priority and batch report pods as low, the scheduler will automatically free resources for the payment service.

Kubernetes also classifies pods into QoS tiers based on requests and limits. Pods with Guaranteed (requests = limits) are the last to be evicted, providing the strongest protection.

5. Elastic reservation & prediction

Static reservation can waste resources during off‑peak periods. Elastic reservation adjusts the reserved amount based on historical traffic patterns and forecasts, scaling up before a predicted surge and scaling down afterward.

This dynamic approach reduces idle capacity while still meeting deterministic guarantees for predictable loads.

Cost of reservation – the inevitable trade‑off

Reservation locks resources, and locked‑idle resources are wasted. For example, reserving 200 CPU cores for a payment service that only uses 40 leaves 160 cores idle, lowering overall utilization.

The core dilemma is:

More reservation → higher safety, lower utilization.

Less reservation → higher utilization, lower safety.

The solution is tiered reservation: give critical paths a Guaranteed level, while less‑time‑sensitive workloads stay in the shared pool.

Recovering idle capacity: overcommit and offline mixing

Overcommit (over‑allocation) allows the sum of requests to exceed physical capacity, e.g., declaring 130 CPU requests on a 100‑core node, assuming not all will peak simultaneously. Properly tuned overcommit improves utilization but can cause eviction if the assumption fails.

Offline mixing (co‑location) borrows idle reserved capacity for batch jobs during off‑peak hours and instantly reclaims it when online traffic returns. Large‑scale operators like Alibaba and Google have raised CPU utilization from ~10 % to >40 % using this technique.

The challenge is ensuring the hand‑off happens within milliseconds and that batch workloads do not interfere with latency‑sensitive services, requiring kernel‑level isolation and sophisticated schedulers.

Observability of reservation

Effective reservation requires continuous monitoring of two metrics: the reserved amount ( requests) and actual usage. A large gap indicates waste; frequent hitting of limits signals under‑reservation. Data‑driven adjustments keep the system in the “sweet spot” of safety and efficiency.

Reservation should also be coordinated with autoscaling and pre‑warming mechanisms so that the reserved pool can instantly handle spikes.

From luck to deterministic supply

In summary, the evolution from a best‑effort shared pool to a tiered reservation model—combining quotas, reserved instances, isolation, priority preemption, elastic prediction, overcommit, and offline mixing—delivers both high guarantee for critical paths and high overall utilization, which is essential for sustaining ten‑million‑QPS workloads.