Why Kafka Achieves Million‑Level Throughput: Sequential Writes, mmap, and Zero‑Copy

Kafka stores messages on disk, yet it delivers extremely high throughput—often millions of writes per second—even on modest hardware. The article analyzes the reasons behind this speed, focusing on write and read optimizations.

1. Writing Data

Kafka writes all incoming messages to disk using two key techniques: sequential writes and memory‑mapped files (mmap).

Sequential writes avoid random I/O, which is costly on mechanical disks; sequential I/O can approach memory speeds.

Linux optimizations such as read‑ahead, write‑behind, and disk caching further improve performance.

Advantages of using disk over JVM heap include avoiding large GC pauses, lower memory overhead, and retaining data after a cold start.

Each partition is a single file; new messages are appended to the file’s end. Offsets are maintained per consumer in Zookeeper, allowing the broker to be unaware of consumer state.

Data retention is handled by two policies: time‑based and size‑based retention, as described in Kafka’s configuration documentation.

Memory‑Mapped Files (mmap)

Kafka maps partition files into the process address space, letting the OS manage paging. Writes go to the mapped memory and are flushed to disk when the producer calls flush or when the OS decides. The producer’s producer.type setting controls sync (flush immediately) versus async (return without flushing).

2. Reading Data

Kafka accelerates reads with several mechanisms:

2.1 Zero‑Copy via sendfile

Traditional read/write involves multiple copies: disk → kernel buffer → user buffer → socket buffer → protocol engine. The sendfile system call reduces these copies by moving data directly from the file descriptor to the socket within the kernel, eliminating user‑space copying and context switches. This dramatically speeds up file transmission and is widely used in web servers such as Apache and Nginx.

2.2 Batch Compression

Instead of compressing each message individually, Kafka compresses batches of messages, reducing network I/O while keeping CPU overhead low. Supported compression algorithms include Gzip and Snappy, and Kafka can store compressed batches recursively until the consumer decompresses them.

3. Summary

Kafka’s performance stems from treating logs as sequential files, using mmap to let the OS handle paging, employing zero‑copy sendfile for reads, and applying batch compression to minimize network traffic. These design choices allow Kafka to sustain high write and read throughput with modest hardware.