From Synchronous to Asynchronous Sending: Achieving 10 Million QPS with Kafka Producer

1. The thread‑blocked order path

During a large‑scale load test the TPS of the order entry was increased from 20k to 120k, causing latency to jump from 30 ms to 1.2 s. The JVM thread dump showed hundreds of business threads stuck in KafkaProducer.send(...).get(), then FutureRecordMetadata.await(), then Object.wait(). The business thread was waiting for a single message ACK, turning the order path into a bottleneck.

2. Real cost of synchronous send

In synchronous mode the producer does three steps: local CPU work, network RTT, and broker write‑ack. The CPU part is <5 % of total latency; the RTT and broker ack together exceed 95 %. A single thread can process at most 1/RTT requests (≈500 TPS for 2 ms RTT). To reach 100k TPS you would need ~200 blocked threads, consuming memory and causing cache thrashing and context‑switch overhead.

When the broker experiences a hiccup (leader switch, GC, disk pressure) the delay is propagated to the business threads, amplifying a 200 ms broker pause into seconds of request timeout and possible circuit‑breaker activation.

Synchronous send also prevents batch and compression benefits because linger.ms never fires while the thread is blocked on the ACK.

3. What asynchronous send changes

Async send separates “submit message” from “wait for ACK”. The business thread only enqueues the record into the Accumulator; a dedicated Sender thread handles network I/O, broker writes and callbacks. This removes the RTT bound from the business thread, allowing a single thread to achieve >10 k TPS (CPU processing <100 µs). Throughput is now limited by local CPU time, not network latency.

With async send the batch size and compression become effective: the Sender aggregates records according to batch.size and linger.ms, achieving 3‑6× compression and reducing broker request count by orders of magnitude.

The latency distribution also flattens: p99 latency is now limited by network rather than thread‑pool queuing, making it easier to control.

4. Async API variants

Callback style: send(record, Callback) – lightweight but the callback runs in the Sender thread, so heavy work must be avoided.

Future/Promise style: returns a Future that can be composed with thenApply, thenCombine, exceptionally, allOf. Adds 50‑100 µs per message for object allocation.

Reactive style: models sending as a Flux with built‑in back‑pressure. Powerful but higher learning curve and harder debugging.

Choosing a variant should be based on the business’s priority among per‑message cost, back‑pressure handling, and error semantics.

5. Delivery semantics

Fire‑and‑forget ( send(record)) – suitable for logging or tracing where occasional loss is acceptable. Requires monitoring of record‑send‑rate, record‑error‑rate, buffer‑available‑bytes.

At‑least‑once – enable idempotence ( enable.idempotence=true) and configure retries and retry.backoff.ms. Guarantees delivery but may produce duplicates; consumer must be idempotent.

Exactly‑once – Kafka 0.11+ provides transactional producer with ProducerId + Sequence Number. Increases init latency, adds 30‑50 % write latency, and requires consumers to read with read_committed. Usually not needed if idempotence suffices.

6. Accumulator and back‑pressure

The Accumulator size is controlled by buffer.memory (default 32 MB). When the buffer fills, send() either blocks up to max.block.ms (default 60 s) or throws TimeoutException / BufferExhaustedException. Most production incidents stem from a full Accumulator, not a broker outage.

Effective back‑pressure should be applied at the business entry point, preventing the Accumulator from reaching its hard limit. Metrics from the Producer (send rate, error rate, buffer usage) must be fed to an observability platform.

7. Evolution roadmap

100k QPS – replace send().get() with async API, enable idempotence, add retry, and apply a modest rate limiter. Throughput improves 30‑50 %.

1 M QPS – tune linger.ms (10‑20 ms), batch.size (64‑128 KB), enable compression (LZ4/Snappy), keep max.in.flight.requests.per.connection ≤ 5.

10 M QPS – shard producers by partition or topic, run multiple producer instances per CPU core, expose per‑instance metrics, and enforce explicit upstream rate limiting.

100 M QPS – decouple producer into a sidecar or proxy service, accessed via Unix socket or gRPC, so the business process never runs producer code.

8. Hidden pitfalls of async

Process shutdown may lose in‑flight messages; call producer.close() or producer.flush() in a shutdown hook (≈30 s timeout).

In‑flight retries can reorder messages; mitigate with idempotence or set max.in.flight.requests.per.connection=1.

Heavy work in callbacks blocks the Sender thread; callbacks should only count, enqueue, or log.

Observability requires six metrics (e.g., record‑send‑rate, record‑error‑rate, buffer‑available‑bytes, batch‑size‑avg, latency‑p99, sender‑thread‑cpu) and corresponding alerts.

9. Final take‑away

Switching from synchronous to asynchronous sending is not merely an API change; it shifts complexity from the business thread to the messaging layer and demands coordinated responsibility among development, architecture, and SRE teams.