Why Kafka Achieves Such High Throughput

Kafka is a widely used distributed event‑streaming platform originally developed at LinkedIn and now maintained by the Apache Software Foundation. It is positioned as a high‑throughput, low‑latency system for building real‑time data pipelines and streaming applications.

1. Sequential Disk Writes Kafka appends all messages to the end of log files, performing pure sequential writes. Because sequential writes avoid disk seek overhead, they can be as fast as or faster than random memory writes, eliminating the performance penalty of random I/O.

2. Zero‑Copy Transfer In a traditional data path, data moves from disk to kernel buffers, then to user buffers, and finally to the socket buffer, incurring multiple kernel‑user switches and copies. Kafka uses zero‑copy, sending data directly from the kernel’s page cache to the network interface, bypassing user space. This reduces context switches and CPU work, allowing the CPU to focus on connection handling and protocol processing.

3. Batch Processing Producers buffer messages in memory and send them as batches (e.g., when the batch reaches 16 KB or after 10 ms). Consumers also pull messages in batches. This dramatically reduces the number of network requests, improving overall throughput.

4. Partition‑Level Parallelism A topic is divided into multiple partitions, each hosted on different brokers. Producers can write to many partitions concurrently, and consumers can read from multiple partitions in parallel, allowing the cluster to fully utilize all machines and achieve high parallel processing capacity.

5. Operating‑System Page Cache Kafka writes incoming messages to the OS page cache and acknowledges the producer immediately, treating the write as a fast memory operation. The actual disk flush occurs asynchronously in the background. Because the cache resides outside the JVM heap, it avoids costly Full GC pauses that could otherwise stall the broker.

6. Message Compression Before sending, producers compress a batch of messages. Brokers store the compressed batch without decompressing it, and consumers decompress the batch locally. This reduces I/O volume and shifts CPU‑intensive compression work to producers and consumers, lowering broker CPU load.

In summary, Kafka’s high throughput is the result of several synergistic design decisions: zero‑copy reduces system‑call overhead, sequential writes maximize disk efficiency, batching and compression improve network and I/O utilization, partitioning enables parallelism, and page‑cache usage avoids JVM GC stalls.