Kafka Performance Design: Sequential I/O, Page Cache, Zero‑Copy, and Partition Segmentation

Kafka is a ubiquitous messaging middleware in the big‑data ecosystem, widely used for real‑time data pipelines and stream processing applications. Although it persists messages to disk, it delivers high performance, high throughput, and low latency, often handling tens of thousands to millions of messages per second.

Sequential Read/Write – Kafka writes messages by continuously appending to the end of log files, avoiding random disk writes. Sequential disk I/O is orders of magnitude faster than random I/O, and modern operating systems heavily optimize this pattern, which dramatically boosts write throughput.

Page Cache – Kafka relies on the OS page cache instead of JVM heap memory. This reduces object overhead, avoids garbage‑collection pauses, and benefits from OS‑level optimizations such as write‑behind, read‑ahead, and automatic flushing. The cache persists across process restarts, eliminating the need to rebuild in‑process caches.

Zero‑Copy – By using the Linux sendfile system call, Kafka can transfer data directly from the page cache to the network NIC, eliminating multiple memory copies and context switches. This reduces the four copy operations and two context switches of a traditional path, resulting in much higher consumer throughput.

Partition and Segment Design – Messages are organized by topic, then by partitions, each mapped to a directory on a broker. Within a partition, data is stored in sequential segments, each with an accompanying .index file. This layout enables efficient range reads, parallelism, and fast lookup.

Conclusion – Combining sequential I/O, OS page cache, zero‑copy transmission, and a partition‑segment storage model (with indexing) gives Kafka its characteristic high performance, high throughput, and low latency, turning large‑capacity disk storage into an advantage rather than a bottleneck.