Resource Profiling at 10M QPS: From Coarse Averages to Fine-Grained Capacity Decisions

Introduction: The Problem with Averages at Scale

At ten-million-QPS scale, coarse averages hide structural capacity risks. A real example: after scaling out a core service, new machines showed 35% average CPU while old machines hit 70%, yet per-instance QPS looked even. Investigation revealed different CPU generations, NUMA topology, and a heavier request mix on new nodes. The capacity platform reported 45% headroom, but some instances were already near latency inflection points. As systems span multiple instance types, availability zones, versions, and complex request patterns, simple averages conceal the differences that matter.

What a Resource Profile Actually Captures

A useful resource profile connects four elements:

Supply characteristics : CPU, memory, disk, network, accelerators, container limits, and topology constraints.

Load characteristics : request types, data sizes, read/write ratios, cache hit rates, concurrency patterns, and burstiness.

Consumption curves : how CPU, memory, I/O, connections, queues, and background tasks vary with load.

Service boundaries : the sustainable throughput under target latency, error-rate, and resilience constraints.

Missing any piece yields an asset inventory, a monitoring snapshot, or a peak-throughput stunt — not a capacity profile. For example, the same 16-core instance can handle vastly different safe QPS for 4 KB cache hits versus 200 KB requests requiring deserialization and authorization. A profile must state conditions explicitly: "Under 80% query mix, P99 < 80 ms, cache hit rate ≥ 96%, with single-instance failure headroom, safe throughput ≈ 16,000 standard work units/sec."

Why Averages Fail at Critical Moments

2.1 Same Spec ≠ Same Capability

Identical cloud instance types can differ in CPU generation, frequency scaling, memory bandwidth, and virtualization contention. Containers add CPU quota, pinning, NUMA effects, shared cache, and noisy neighbors. Two instances both allocated 8 vCPUs may show dramatically different utilization curves under identical load; averaging them masks the weaker node's risk.

2.2 Same Request Name ≠ Same Cost

Treating QPS as work units works only for uniform workloads. In search, recommendation, risk-control, or gateway services, a simple query may read one cache key while a complex query fans out to 30 downstreams, scans thousands of candidates, and runs model inference. Both count as 1 QPS but consume vastly different CPU, memory bandwidth, and connections. A shift from 5% to 18% complex requests can raise resource usage without any QPS change, misleading capacity models into diagnosing "instance performance degradation."

2.3 Time Averaging Erases Bursts and Queuing

A 1-minute average of 10k QPS could be steady 10k/sec or 8k/sec for 50 seconds then 20k/sec for 10 seconds. The latter saturates thread pools, connection pools, and queues, causing timeouts even after traffic subsides. Minute-level averages produce postmortems where "monitoring showed no capacity breach, but users timed out."

2.4 Cluster Averaging Erases Local Hotspots

In sharded systems, 55% cluster disk usage doesn't guarantee the hottest shard isn't IOPS-bound; 95% overall cache hit rate doesn't mean a tenant's hot key isn't pegging a single core. Profiles must resolve to the actual constrained unit: instance, shard, queue, tenant, or failure domain.

Averages describe the whole; capacity decisions must see the part that actually bottlenecks.

Building a Usable Coarse-Grained Profile First

3.1 Bucket by Service × Spec × Region

Start by splitting "one number for the whole service" into buckets of service version × instance spec × deployment region. This eliminates most obvious heterogeneity errors. Avoid over-splitting: each bucket needs enough samples for stable P99 and throughput curves. Low-sample buckets can merge with similar specs or use offline stress-test priors until online data accumulates.

3.2 Choose an Interpretable Work Unit

The simplest unit remains QPS, but its applicability must be explicit. When request costs vary, define a standard work unit. For instance, designate a baseline query as 1 work unit, an aggregation query as 3.2, and a write as 2.5. A minute with 6,000 simple, 1,000 aggregation, and 500 write requests yields ~9,800 work units/sec, not 7,500 QPS. This conversion need not be perfect initially; it only needs to be closer to real consumption than "all requests cost the same."

3.3 Define Safety Boundaries via SLO Inflection Points

Profile capacity limits should not be CPU 100% or stress-test peaks. Instead, trace the load curve to find where business metrics (e.g., P99 latency) begin nonlinear deterioration. If P99 stays at 55 ms at 18k work units/sec, rises to 79 ms at 20k, and jumps to 148 ms with growing queues at 22k, the safe boundary lies between 18k–20k. Then apply production discounts: online environments run ~8% slower than stress tests, and a 20% failure reserve is prudent. The final number published to the capacity platform separates "can run" from "can reliably deliver."

3.4 Attach Validity Metadata

Profiles expire as versions, compilers, dependencies, data distributions, and infrastructure change. Every profile must record: sample time range, service and dependency versions, instance spec and runtime limits, load structure, data volume and hit rates, confidence level and sample count, and last calibration timestamp. Expired profiles don't instantly invalidate but should trigger reduced confidence, prompting re-test or wider safety margins.

From QPS to Workload Fingerprints

4.1 Key Load Dimensions

Different systems have different dominant dimensions. Common candidates include: request type and business scenario, request/response/batch size, read/write ratio, downstream fan-out, data scan volume and returned rows, cache hit rate, compression/encryption/serialization methods, model size/token count/candidate set size, long-connection ratio and hold time, tenant tier and burst coefficient. Too many dimensions cause combinatorial explosion; in practice, use performance analysis and correlation to select 3–6 leading features and cluster similar requests into a few load types. For example, a search service can fingerprint by query complexity, filter count, recall shards, and candidate set size; a gateway can group by message size, auth path, protocol translation, and downstream fan-out.

4.2 Convert Workload to Unified Cost

Pick a stable request class as baseline and compute relative weights from CPU time, I/O bytes, memory allocations, and downstream call costs. Let weight of class i be w_i and arrival rate q_i; total work W = Σ(i=1..n) q_i × w_i. W = Σ(i=1..n) q_i × w_i Weights need not be CPU-only. For storage services, disk IOPS and network may dominate; for proxies, connections and message size; for AI inference, GPU memory, token count, and batch efficiency. A more realistic approach maintains separate weight sets per bottleneck dimension: CPU work units, memory-bandwidth work units, I/O work units, network work units, downstream-dependency work units. Final capacity is determined by the dimension that hits its safety boundary first, preventing a single composite score from masking bottleneck shifts.

4.3 Profiles Must Explain Version Changes

After a release, a 12% per-request CPU increase could stem from new features or performance regression. If the profile captures both request structure and version, it can distinguish "traffic got heavier" from "code got slower." A practical comparison: select similar load windows for old and new versions, compute per-work-unit resource consumption. If request fingerprints match but unit cost rises, the change is likely version-induced. The capacity platform can then re-estimate required instances before full rollout, instead of discovering headroom shortage after CPU alerts fire.

Fine-grained profiling means finding the few dominant variables that explain resource consumption changes.

Connecting Offline Stress Testing with Online Observation

5.1 Offline Stress Tests Provide Comparable Baselines

Controlled environments fix datasets, request mixes, versions, and background tasks, then ramp load to produce full throughput-latency curves. They excel at answering: relative performance across instance types, per-request cost change in new versions, which resource bottlenecks first, where latency inflection occurs, and how request-mix shifts affect safe capacity. However, stress environments are cleaner: stable dependency latency, no noisy neighbors, easier cache warm-up. Results must carry a production discount factor; they cannot be used raw as production capacity.

5.2 Online Observation Corrects for Environment and Real Load

Online data reveals long tails, bursts, data skew, dependency jitter, and background tasks. Calibration windows must be stable: consistent version and config, no releases/scaling/failovers, normal error rates, warmed caches and connections, sustained load, and healthy key dependencies. For each profile bucket, compare predicted vs. actual consumption. If the model chronically underestimates CPU by 8%, lower that bucket's safe capacity or adjust work-unit weights; if distortion appears only when a specific request class grows, enrich the load fingerprint.

5.3 Feedback Loop Prevents Profile Staleness

Profile updates must not chase minimal error at the cost of decision stability. If the model changes safe capacity daily, autoscaling will thrash. Enforce minimum sample sizes, change caps, and observation periods; critical profiles require replay validation. Maintain three values per bucket:

Observed value : recent-window estimate of true capability.

Published value : the stable capacity currently used by the platform.

Candidate value : newly computed capability awaiting verification.

Candidate values graduate to published only after multiple validation cycles, allowing learning while preventing a single anomalous sample from corrupting global capacity.

Fine-Grained Profiles Must Land on the Bottleneck Resource

Many platforms emit a single "health score" or "capacity score." A single number simplifies dashboards but cannot guide action. CPU, memory, disk, network, and dependency quotas require different mitigations; profiles must preserve bottleneck semantics.

6.1 CPU Profiles: Compressibility and Queuing

CPU utilization ≠ CPU pressure. Combine throttle time, run-queue length, context switches, single-core hotspots, and per-work-unit CPU time. For event-loop or single-threaded hotspot services, 45% average CPU may already be single-core bound. For horizontally parallel stateless compute, 70% may remain stable, but watch for latency curve inflection.

6.2 Memory Profiles: Resident, Burst, and Reclamation

Memory splits into base resident, concurrency-dependent, data-volume-dependent, and short-lived allocations. A flat "8 GB per instance" estimate conflates cache growth, heap objects, page cache, and off-heap memory. Profile with a formula:

M = M_base + C × m_concurrency + D × m_data + M_burst

where C is concurrency, D is data scale. Safety boundaries must subtract GC overhead, kernel reserves, and burst buffers. For containers, respect cgroup limits and working set — host free memory is not a substitute.

6.3 I/O Profiles: Pattern Matters, Not Just Bandwidth

Writing 100 MB/sec sequentially in large blocks stresses storage differently than random small writes. Disk profiles must capture IOPS, throughput, queue depth, read/write ratio, block size, tail latency, and background compaction. Databases and log systems must separate foreground requests from background tasks (compaction, checkpoint, backup, snapshot, replica sync) that periodically seize I/O. Modeling only foreground QPS makes capacity suddenly inaccurate when background jobs kick off.

6.4 Network Profiles: Packet Rate and Failover

Network is not just Gbps. Small-packet workloads hit PPS, soft-interrupt, or connection-tracking limits first; large-packet workloads hit bandwidth limits. Cross-AZ and cross-region traffic adds link quality, cost, and failover capacity concerns. Network profiles should distinguish inbound, outbound, packet rate, connection establishment, active connections, and retransmits. Under N-1 scenarios, estimate whether remaining links can absorb migrated traffic.

A profile must tell you why you're near the boundary; otherwise it can only alert, not prescribe capacity actions.

At 10M QPS, Failure Domains and Topology Matter

7.1 From Instance Profiles to Failure-Domain Profiles

Instance profiles describe single-unit capability; failure-domain profiles describe the capacity of an AZ, datacenter, or cluster after losing part of its supply. Simple summation fails: 100 instances × 10k work units = 1M theoretical, but if 20% of shard data is hot only on a subset, egress links cap at 850k, and dependency quotas cap at 900k, the true AZ capacity is 850k. Aggregated profile takes the minimum across compute, memory, network, dependency, and data constraints, factoring in scheduling reachability:

Capacity_zone = min(C_compute, C_memory, C_network, C_dependency, C_data)

7.2 Make N-1 Part of the Profile

Healthy steady-state capacity ≠ healthy failure capacity. Profiles should precompute common failure scenarios: lose one instance, lose one shard replica, lose one AZ, lose one egress link, lose a critical dependency to its minimum quota, lose scaling capability temporarily.

At 10M QPS, a 5% capacity error equals hundreds of thousands of QPS. Failover isn't evenly redistributing traffic; data locality, session stickiness, tenant isolation, and link bandwidth constrain migration paths. Fine-grained profiles must align with scheduling topology. If the capacity platform computes per-AZ but the load balancer can only migrate within a small cluster, the platform's "global headroom" is unusable.

7.3 Separate Sellable Capacity from Reserved Capacity

Total safe capacity splits into: normal business sellable capacity, failure takeover reserve, release and scaling operation reserve, prediction error buffer, and temporary event/burst allowance. Different business tiers may use different buffer strategies, but the same failure reserve cannot be sold twice. The profile platform must track which policy consumes each capacity slice to prevent multiple systems from believing they own the same 20% margin.

Getting Profiles Into the Decision Loop, Not Just Dashboards

8.1 Scheduling: Route Requests to Better-Matched Resources

When the platform knows different instance types have different efficiencies for different load types, it can route more intelligently: memory-bandwidth-sensitive requests to high-bandwidth pools, long connections to instances with ample connection capacity, batch traffic away from latency-sensitive pools. Scheduling should not chase equal CPU percentages across heterogeneous instances; it should equalize each instance's relative distance to its own safety boundary.

8.2 Autoscaling: From Instance Count to Work Deficit

Traditional rule: "CPU > 60% → add 20% instances." Profile-driven: forecast work-unit deficit for the next window, then convert to instance count using the target spec's safe capacity under current load structure. Example: expected 300k work units/sec increase in 15 minutes; target instance safe capacity 12k work units/sec; scale-up success and warm-up factor 0.9 → need ceil(300,000 / (12,000 × 0.9)) = 28 instances. N = ceil(300000 / (12000 × 0.9)) = 28 If only older instances are available (safe capacity 8,500), the calculation yields more instances. Scale quantity derives from real capability, not "one instance = one instance."

8.3 Releases: Turn Performance Changes Into Capacity Gates

During canary, compare old vs. new version per-work-unit cost under similar load. If the new version's safe capacity drops beyond a threshold, the release system can demand capacity补齐, rollback, or extended observation — before full rollout triggers alerts.

8.4 Degradation: Drop the Most Expensive Work First

Knowing which requests are costly lets the system shed high-cost non-critical traffic first when capacity tightens. Cutting 5% of expensive requests may be more effective and less damaging to core user paths than randomly rate-limiting 15%.

A profile's value is measured by decision quality, not by tag count or model complexity.

Boundaries and Governance Cost of Granularity

9.1 Avoid Dimension Explosion

If version, spec, region, request type, tenant, and data volume each have dozens of values, a Cartesian product creates massive sparse buckets. Use a hierarchical model:

Global base capability — default values.

Spec and version corrections — eliminate major heterogeneity.

Load-type corrections — explain request cost differences.

Region and topology corrections — express environmental discounts.

Special profiles for a few key tenants or hotspots.

Low-sample combinations inherit upper-level results; high-sample combinations learn their own correction factors. This preserves coverage while avoiding false precision for rare combos.

9.2 Attach Confidence to Every Number

A capacity value measured 20 times across multiple load bands should not carry the same weight as a single short online observation. Published profiles must carry: sample size and coverage window, prediction error, whether inflection point was covered, whether online-calibrated, time since last validation, and whether key features have drifted. Low confidence → larger safety discounts or disable auto-shrink. Admitting "uncertain" is more engineering-valuable than pretending precision.

9.3 Keep Models Explainable

Complex ML models may reduce average error, but on-call engineers need to know why the system recommends 40 instances instead of 20. Profiles must surface: which resource dimension is binding, which request class drives cost, which spec/version curve was used, how much failure and prediction reserve was subtracted, and what fallback rule applies if the judgment is wrong. Explainability makes automated decisions auditable and rollback-able — not a dashboard ornament.

9.4 Enforce Data Quality Gates

Profiles depend on service tags, metric definitions, tracing sampling, and asset data. Any misalignment yields precisely wrong conclusions. Continuously verify: instance spec matches container limits; QPS, work units, and resource metrics are time-aligned; release, failure, and stress-test windows are correctly labeled; missing values and sampling rates are stable; request classification covers new APIs; metric resets, counter overflows, and aggregation methods are correct. Profile updates pass data-quality gates first; on anomaly, retain previous published value — don't let dirty data drive autoscaling.

Four Phases: From Spreadsheet to Runtime Capability

Phase 1 — Unify Caliber: Every service answers: "Under what SLO does this spec's single instance have X safe capacity?" No complex algorithms yet.

Phase 2 — Load Structure and Version Changes: Pick a few dominant features, convert QPS to comparable work units, establish canary performance comparison.

Phase 3 — Multi-Dimensional Bottlenecks + N-1/Topology: Stop using one composite score for CPU, memory, I/O, network, and dependency limits. Fold N-1, data locality, and scheduling topology into aggregation.

Phase 4 — Full Closed Loop: Profiles feed prediction, autoscaling, scheduling, release gating, and degradation; every execution result feeds back to calibrate profiles. Automate from low-risk scenarios: suggest → semi-auto → full auto for stable cases.

Evaluate with three metrics:

Accuracy: Are resource consumption and safe capacity prediction errors decreasing?

Decision Effect: Are insufficient scaling, wasteful scaling, and local overloads decreasing?

Governance Cost: Are model maintenance, data collection, and manual calibration costs controlled?

If profiling dimensions double but capacity error and incident count don't improve, that refinement cycle produced no return. Conversely, adding just "request size" as a feature and significantly cutting prediction error makes it worth prioritizing.

Conclusion: Precision Is Not the Goal — Trustworthiness Is

Coarse profiles aren't original sin. Early-stage systems with uniform instances, simple requests, and simple failure domains correctly choose per-instance QPS plus a safety factor as the lowest-cost right answer. As systems grow, heterogeneity in hardware, versions, load, and topology turns average errors from a few percentage points into real risks. At that point, augment profiles layer by layer: first separate specs, then identify load costs, then preserve multi-resource bottlenecks, finally integrate failure reserves and online calibration into decisions.

The true measure from coarse to fine is whether the profile can stably explain resource consumption, predict safety boundaries, and let capacity actions be verified.

If you could add only one profiling dimension for a core service today — instance spec, request type, data scale, or failure domain — the answer isn't universal. Start from the most frequent, highest-cost capacity misjudgment you currently face.