Why Moving from Real‑Time to Batch Is Essential for Scaling to Tens of Millions QPS

Why Single‑Request Processing Fails at Tens of Millions QPS

Each request incurs a fixed cost that is largely independent of payload size (system‑call context switch, network handshake, lock acquisition, B+‑tree index lookup, WAL fsync, RPC serialization). If the fixed cost is 100 µs and the variable cost is 1 µs, at 1 M QPS the fixed overhead totals 100 s per second, which can be tolerated. At 10 M QPS the fixed overhead becomes 1 000 s, overwhelming the system.

The Essence of Batching: Diluting Fixed Costs

Batching groups many requests so the fixed cost is paid once per batch. For 100 requests, a 100 µs fixed cost is shared, giving 1 µs per request, while the variable cost for the batch is 100 µs. Effective per‑request cost drops to 2 µs – a fifty‑fold reduction.

Where Batching Applies: From Client to Storage

Batching is not limited to databases.

Network layer: TCP’s Nagle algorithm coalesces small packets; OS calls writev and sendmmsg send multiple buffers in a single system call.

Redis pipelining sends many commands in one round‑trip, eliminating a network round‑trip per command.

Kafka producer batches records based on batch.size (bytes) and linger.ms (time). Disabling these parameters causes a dramatic throughput drop.

Database: inserting 100 rows with a multi‑value INSERT versus 100 single inserts yields tens‑fold throughput gains because the fixed cost of SQL parsing, transaction start/commit, index update, and log fsync is paid once.

Group commit merges many concurrent transaction fsyncs into a single disk write.

When to Trigger Batching: Three Strategies

Size‑based trigger : emit a batch when N items are collected. Guarantees batch size but can cause unbounded latency under low traffic.

Time‑based trigger : emit whatever is collected every T ms. Bounds latency but may produce oversized batches during spikes, risking memory exhaustion.

Hybrid trigger : emit when either N items are ready or T ms have elapsed, whichever comes first. This is the pattern used by Kafka and most mature batch components, balancing throughput, latency, and memory pressure.

Throughput Gains vs. Latency Cost

Batching adds a small waiting time while items accumulate, but overall system latency can still drop. When a system is overloaded, Little’s Law shows that queue length (and thus latency) grows dramatically. By increasing throughput, batching prevents queue buildup, so the added wait is negligible compared to the latency caused by overload.

Batching Is Not Free

Introducing batching creates new complexities:

Partial failures : a batch may contain invalid records. Deciding whether to roll back the whole batch or isolate failures adds logic.

Failure amplification : when a batch fails, many requests are affected simultaneously.

Memory pressure : buffered items occupy memory; without back‑pressure, buffers can overflow and cause OOM.

Latency spikes : large batches create processing bursts that appear as P99‑P50 gaps.

Ordering and consistency : parallel batch processing can reorder events; atomicity decisions must be made up‑front.

Scaling Thresholds

At ~100 k QPS, single‑request processing is simple and the aggregate fixed cost fits within system capacity. Around 1 M QPS, hot paths (e.g., frequent DB writes) start to dominate, making batching a key optimization. At 10 M QPS, the total fixed cost is so large that the system cannot survive without systematic batching across the entire request‑to‑storage pipeline. The shift from per‑request handling to pervasive batching separates a scalable architecture from one that collapses.

Illustrative Example

Original workload: write each telemetry record to the database immediately. After changing the logic to “collect a batch then write once”, throughput increased by dozens of times and latency dropped, demonstrating that the bottleneck was the per‑request fixed cost, not the data volume.

Key Takeaways

Batching’s benefit is proportional to the magnitude of fixed cost in a given operation.

Optimal batch size results from a trade‑off among fixed‑cost amortization, memory limits, and latency tolerance; it must be determined by measurement.

Hybrid size‑/time‑based triggers provide bounded latency while protecting memory.

Batching is essential for high‑density traffic (hundreds of thousands to tens of millions QPS) but adds complexity that must be managed.