Cost Optimization at Ten‑Million QPS: Turning Ignored Expenses into Core Design

During a routine quarterly review the team celebrated raising availability from 99.95% to 99.99% and proving the system could handle three times the peak load. The CFO then displayed a cloud bill that had risen 40% quarter‑over‑quarter while business orders grew only 12%. Monitoring showed cluster‑wide CPU utilization stuck between 11% and 14%, revealing that cost had been treated as a post‑mortem finance line rather than a design concern.

Why Cost Is Often Ignored Early

In the early stages of a startup, survival depends on keeping the service alive. Engineers prioritize performance and availability because failures are visible and costly, while cost is a monthly finance issue that seems unrelated to daily work. This mindset leads to a "more is better" culture: services request double the CPU they actually need to guard against spikes, resulting in chronic over‑provisioning. When the system is small, the extra spend is a negligible fraction of the total budget, so fine‑grained cost control feels unnecessary.

The hidden costs manifest in three ways:

Low utilization : average CPU usage stays at 10%–20%, meaning ten machines are bought but only one or two are doing real work.

Peak‑based provisioning : resources are locked for the highest daytime load, leaving machines idle during night‑time troughs.

Cost invisibility : without unit‑cost metrics or allocation tags, engineers have no sense of how much a single request or service costs, so no one feels responsible for optimization.

Why Cost Becomes Core at Ten‑Million QPS

Scale magnifies every inefficiency. At a few hundred machines, a 1% utilization gain saves a few hundred thousand yuan a year; at tens of thousands of machines, the same 1% saves millions. In high‑throughput businesses, infrastructure cost can become the second‑largest expense after labor, directly affecting unit economics and the viability of scaling.

Step 1: Make Cost Observable

The first step is not to cut machines but to measure cost. Three artifacts are required:

Unit cost : divide total spend by business volume (e.g., cost per 10,000 requests or per order) to turn a big bill into a metric engineers can act on.

Cost allocation : tag workloads by service, team, namespace, etc., and use tools such as Kubecost or OpenCost to split node cost by actual CPU/memory usage.

Cost dashboard : combine unit cost, allocation results, and utilization watermarks into a continuously refreshed view that highlights outliers.

When cost becomes visible, engineers start asking "why is my service's unit cost high?" instead of waiting for finance to raise the issue.

Step 2: Increase Utilization

Spec‑matching (Right‑sizing)

Teams habitually over‑request resources (e.g., asking for 4 CPU cores when the service peaks at 2). Right‑sizing uses historical usage data to set realistic requests and limits. Kubernetes VPA or cloud provider compute optimizers can recommend appropriate instance types. The trade‑off is to keep a safety margin to avoid throttling during spikes.

Mixed Workloads, Elastic Scaling, and Over‑commit

Offline batch jobs can share the same machines used by online services during low‑traffic periods, raising overall utilization from ~10% to 40%+. Elastic scaling automatically expands capacity for peaks and shrinks it for troughs, correcting the "peak‑only" provisioning model. Over‑commit (or "selling") relies on the statistical fact that not all services hit their peak simultaneously; it requires robust monitoring and eviction policies to avoid contention.

Bin‑packing

Bin‑packing schedulers place new pods onto nodes that already have some free CPU/memory, packing workloads tightly before launching new nodes. This improves packing density at the cost of reduced fault isolation compared to uniform spreading.

Step 3: Use Cheaper Resources

After utilization is high, the next lever is price. Three resource classes are combined:

Reserved instances / Savings Plans : commit to a baseline level of usage for 1–3 years to obtain 30%–40% discounts.

Spot (preemptible) instances : take advantage of idle cloud capacity at 10%–20% of on‑demand price, suitable for interruptible workloads such as stateless services, batch jobs, or CI tasks. Implement graceful shutdown handling and diversify instance types and zones.

ARM or custom chips : newer cloud‑provider CPUs often deliver lower cost per compute unit for the same performance; migration mainly requires recompilation.

The resulting cost structure is: baseline load on reserved capacity, variable load on on‑demand elasticity, and interruptible load on Spot.

Step 4: Architectural Cost Reduction

Design choices that were originally made for performance also cut cost:

Caching : a well‑designed cache reduces backend traffic, allowing a smaller database and service cluster.

Asynchronous processing : moving synchronous calls to message queues smooths spikes, letting capacity be sized for average load instead of peak.

Cold‑hot data tiering : store infrequently accessed data in cheap object or archival storage.

Efficient serialization/compression : tighter formats and compression lower bandwidth and storage fees.

Read‑write splitting : offload reads to replicas, reducing the size of the primary database instance.

These patterns illustrate that good architecture and low cost are two sides of the same coin.

Step 5: Shutdown and Reclaim Idle Resources

Long‑running systems accumulate "zombie" resources: unused test instances, orphaned databases, forgotten load balancers, and stale snapshots. Automated reclamation scans for zero‑traffic resources, notifies owners, and decommissions them after a grace period. Non‑production environments can be scheduled to scale down or shut off outside working hours, cutting a large fraction of their bill.

Step 6: Guardrails – Cost Optimization Must Not Break SLA

Pure cost‑cutting can degrade latency, increase tail‑latency spikes, or cause service outages. Guardrails include:

Set utilization caps that leave headroom for traffic spikes.

Limit Spot usage to a safe proportion of total capacity and ensure interrupt‑tolerant design.

Define minimum capacity thresholds and rapid scaling policies for sudden load.

Validate every cost‑saving change in a canary or staging environment before full rollout.

Step 7: Make Cost a Culture – FinOps

Tools are only enablers; the real shift comes from embedding cost awareness into daily engineering practice:

Cost in design reviews : alongside performance and availability, ask "what is the cost structure and unit cost?"

Engineers own their service cost : each team sees its own spend on the dashboard and tracks unit‑cost trends.

SLOs tied to cost : choose reliability targets that balance business value against exponential cost increases for higher availability.

Budgets and alerts : set cost budgets per team or service and trigger alerts on overruns.

When cost awareness becomes a shared responsibility, the technical measures described earlier become routine rather than occasional cost‑cutting projects.

Conclusion

The article walks through a clear path: first make cost observable (unit cost, allocation, dashboard), then extract idle compute through right‑sizing, mixed workloads, elastic scaling, over‑commit, and bin‑packing; next compress the price curve with reserved capacity, on‑demand elasticity, and Spot; then let architecture (caching, async, tiering, compression, read‑write split) inherently save money; finally, institutionalize FinOps so cost is discussed in design reviews, engineers own their spend, SLOs consider cost, and budgets trigger early warnings. The key insight is that at ten‑million‑QPS scale, cost optimization is not a side activity but a core design objective that must coexist with performance and availability.