How Kafka Achieves Million‑TPS with Sequential I/O, MMAP, and Zero‑Copy

How Kafka Achieves Million TPS

Kafka is hailed as a "killer app" for big data, widely adopted for its ability to handle millions of transactions per second, making it a cornerstone for data collection, transmission, and storage.

Sequential Disk Read/Write

Both producers and consumers perform sequential reads and writes, which dramatically outpaces random I/O. Sequential access on modern HDDs or SSDs can even exceed random memory reads, so Kafka stores each partition as a file and appends new records to the file tail.

Traditional disks consist of platters, heads, tracks, cylinders, and sectors. Data is organized into blocks, which are groups of sectors; reading a block sequentially avoids costly seek and rotational delays.

Sequential read/write : Accessing records in their logical order on storage. Random read/write : Access time independent of the data's physical location.

Memory Mapped Files (MMAP)

To narrow the gap between disk and memory speeds, Kafka uses memory‑mapped files. MMAP maps a file directly into the process's address space, allowing the OS to handle paging and eliminating user‑space to kernel‑space copies.

Kafka exposes the producer.type setting: when set to sync, data is flushed to disk after each write; when async, the write returns immediately without flushing.

Zero Copy (Zero‑Copy)

Zero‑copy leverages Direct Memory Access (DMA) to move data without CPU copying. In a conventional path, data passes through four stages (disk → kernel buffer → user buffer → socket buffer → NIC). Kafka reduces this to two stages: disk → kernel buffer (DMA) and kernel buffer → NIC (DMA), bypassing user‑space copies.

This reduction cuts transfer time by about 65%, significantly boosting throughput.

Batch Data Processing

Instead of sending one message at a time, Kafka batches messages into files. When a consumer requests data, Kafka can transmit the entire file (e.g., 1 M messages ≈ 10 MB) in roughly one second over a good network, achieving ~1 M TPS.

Batching also enables compression of the whole file, further reducing network I/O.

Summary

(1) Sequential writes to the end of each partition maximize speed.

(2) MMAP maps disk files into memory, allowing disk‑like operations on memory.

(3) Zero‑copy via DMA eliminates intermediate CPU copies, halving data transfers.

(4) Sending whole files with sendfile and compressing them reduces network overhead.