Scaling at Ten‑Million QPS: From Manual to Automatic Autoscaling

Why Manual Scaling Fails at Ten‑Million QPS

During a pre‑Double‑Eleven weekend, an ops team manually expanded a core transaction service to 800 instances at 23:00, only to keep them running idle for a week because nobody dared to shrink them, illustrating four fundamental pain points of manual scaling: inaccurate human capacity estimation, a long decision‑execution chain that misses the golden window, shrink‑lag due to risk aversion, and overnight incidents when no one is on‑call.

Metric‑Driven Scaling: From Resources to Business Indicators

Automatic scaling starts with making the service “feel” its load. The simplest approach uses resource metrics such as CPU and memory (e.g., Kubernetes HPA expands when CPU > 60%). However, resource metrics are lagging and may not reflect true load for I/O‑bound services. Business metrics—QPS, response time, queue length, active connections—directly describe the pressure on the service and are fed to HPA via Prometheus custom‑metrics adapters or to Knative’s KPA.

Scaling Algorithms and Flapping Prevention

Two common algorithms are target‑tracking (desired replicas = current replicas × (current metric / target metric)) and step‑scaling (pre‑defined rules such as “CPU 70‑85% → +2 pods, >85% → +5 pods”). To avoid flapping, three safeguards are applied: a cooldown window after each scaling action, an asymmetric “scale‑fast‑shrink‑slow” rule (fast expansion, delayed contraction), and a tolerance band (e.g., ±10% around the target) that suppresses minor metric oscillations.

Capacity Water Level and Redundancy

Because new instances need lead time (seconds to minutes) for scheduling, image pull, container start‑up, JIT warm‑up, and cache filling, scaling decisions must account for this delay. Setting a headroom—target utilization around 60% instead of 90%—provides buffer space to launch instances before the existing ones are overwhelmed. The optimal water level balances cost (lower utilization wastes resources) against safety (higher utilization risks overload).

Reactive vs. Predictive Autoscaling

Reactive scaling reacts to metric spikes, which inevitably lags behind sudden traffic bursts. Predictive scaling attempts to pre‑provision capacity using two sources: historical traffic cycles (daily peaks, weekly patterns) and an event calendar (planned promotions, product launches). Predictive scaling can miss unexpected spikes and relies on model accuracy, so a hybrid approach—predictive for known patterns, reactive as a safety net—is recommended.

Stateful Service Scaling Challenges

Stateless services scale easily because any instance can replace another. Stateful services face data rebalancing (e.g., shard migration when expanding from 4 to 6 shards), session ownership loss during shrink‑down, and long‑connection migration for IM or push services. Common remedies include externalizing session state to Redis, designing for graceful connection handoff, and performing rebalancing in batches.

Elastic Boundaries and Protection

Automatic scaling can itself become a failure source if unchecked. Protective measures include hard min/max replica limits, coordination with rate‑limiting and degradation mechanisms, and recognizing downstream bottlenecks (databases, caches) as the true capacity ceiling. These safeguards ensure scaling stays within safe operational bounds.

Multi‑Dimensional Scaling: Horizontal, Vertical, and Event‑Driven

Beyond classic horizontal pod autoscaling (HPA), vertical pod autoscaling (VPA) adjusts CPU/memory per pod for workloads that cannot be horizontally split. Event‑driven autoscaling (KEDA) scales based on external event backlogs such as Kafka lag or RabbitMQ queue length, and can even scale to zero for truly idle services, enabling serverless‑style cost savings.

Dynamic Balance as the Essence of Elasticity

The article concludes that there is no silver bullet: pure reactive scaling suffers latency, pure predictive scaling depends on imperfect models, and water‑level choices trade cost for safety. A mature solution combines metric‑driven reactive scaling, predictive pre‑deployment, appropriate headroom, and strict elastic boundaries to continuously balance cost against stability.