Scaling to Millions of QPS: How Batch Consumption Beats Single-Message Processing

Why Single‑Message Consumption Hits a Ceiling

In a real incident, an order service accumulated 12 million pending messages and could only process about 8 000 messages per second despite a 32‑core CPU being only 30 % utilized. Profiling showed each consumer thread repeatedly performed a pull, a DB read, a DB write, and an offset commit for every single message, limiting throughput to roughly 5–10 ms per message (≈100–200 messages per second per thread).

Key insight: the bottleneck is not CPU but the fixed protocol and transaction overhead that cannot be amortized across messages.

What Batch Consumption Changes

Batch consumption does not merely pull multiple messages at once; it also batches the processing and commit steps, turning three separate per‑message operations into per‑batch operations. The number of pipeline stages stays the same, but each stage handles, for example, 500 messages instead of one, turning per‑message costs into per‑batch costs and multiplying throughput by tens of times.

Three Dimensions of Merging

Pull merging: fetching many messages in a single network round‑trip.

Processing merging: executing DB reads/writes and business logic in bulk.

Commit merging: committing offsets for the whole batch at once.

Only merging the pull stage yields modest gains; the real leverage comes from batching DB, RPC, and external calls.

Costs Introduced by Batching

Increased Latency

In single‑message mode, end‑to‑end latency is the sum of the per‑message pipeline (milliseconds). In batch mode, a message must wait until the batch is full, adding queuing delay that is noticeable in latency‑sensitive scenarios such as fraud detection.

Memory and GC Pressure

Fetching 500 × 4 KB messages requires a ~2 MB buffer, plus deserialized objects and intermediate structures, often doubling memory usage. Larger batches keep temporary objects alive longer, causing them to survive the Young GC and trigger Full GC, which can be disastrous.

Failure Amplification

A single message failure can abort the entire batch. For example, inserting 500 orders where the 287th violates a unique index rolls back the whole transaction, wasting the first 286 messages and leaving the rest unprocessed. Re‑nacking the whole batch can cause a live‑lock.

Choosing the Right Batch Size

Batch size is a trade‑off knob between throughput and latency. The article suggests three typical scenarios with starting values, then refines the size using three constraints (e.g., processing time, memory, and poll interval) and taking the minimum, finally reducing by 30 % as a safety buffer.

Kafka’s default max.poll.interval.ms is 5 minutes; if a batch of 10 000 messages takes 8 minutes to process, the consumer is considered dead and a rebalance occurs, wasting the whole batch.

Dynamic Batching

Static batch sizes either waste latency during low load or exhaust memory under peak load. A dynamic approach combines a maximum batch size ( max.batch, e.g., 500) with a maximum wait time ( linger.ms, e.g., 50 ms). Whichever condition is met first triggers processing, keeping latency bounded while still benefiting from batch gains.

More advanced schemes use feedback loops (PID controller or exponential moving average) to adjust batch size based on downstream processing speed.

Handling Batch Failures

Three common strategies are illustrated (images omitted):

Retry the whole batch.

Split the batch and retry only the failing subset.

Move the problematic messages to a dead‑letter queue.

Idempotent business logic is mandatory whenever batch retries are possible; otherwise duplicate processing will occur. The article lists typical idempotency techniques (image omitted).

Dead‑Letter Queue Pitfalls

Do not treat the dead‑letter queue as a trash bin; it still requires monitoring and analysis. Also, only unrecoverable failures (e.g., deserialization errors) should go to the dead‑letter queue, while transient errors belong in a retry queue.

MQ‑Specific Batch Capabilities

Different message brokers support batching to varying degrees. A quick reference chart (image omitted) compares Kafka, RocketMQ, and others. For Kafka, the article stresses three configuration details: max.poll.interval.ms must exceed the worst‑case batch processing time.

Disable auto‑commit ( enable.auto.commit=false) and commit offsets only after successful batch processing.

In ordered consumption, combine partition‑level parallelism with per‑partition batching.

RocketMQ’s ordered consumer uses MessageListenerOrderly and controls batch size via consumeMessageBatchMaxSize. Maintaining order across batches requires careful handling; parallel processing inside a batch breaks order guarantees.

Scaling Path from 100 k to 10 M QPS

The article outlines three stages of evolution, each with different focus points (images omitted). At the 10 M QPS level, a single batch strategy is insufficient; traffic is split by message type to multiple consumer clusters, each tuned with its own batch parameters, turning batch size into a per‑layer SLA tool rather than a global constant.

Recap of the 12‑Step Refactor for the 12 M Backlog

Identify that the bottleneck is fixed per‑message overhead, not CPU.

Introduce batch pulling (≈500 messages) to reduce network round‑trip cost.

Convert DB SELECT/INSERT to batch SQL, dropping per‑row DB cost from ~5 ms to ~0.05 ms.

Batch offset commits and ensure business logic is idempotent.

Add linger.ms + max.batch limits to bound latency.

Set up dead‑letter and degradation strategies to isolate toxic messages.

After these changes, the same 32‑core machine processed ~68 000 messages per second, clearing the 12 M backlog in under 30 minutes without increasing CPU, memory, or I/O usage.

Takeaway: Batch consumption is a paradigm shift for high‑throughput systems, turning per‑message fixed costs into amortized batch costs. It is the right tool for massive QPS scenarios, but the batch size must be tuned to the latency, memory, and failure‑handling requirements of each workload.