Zero-Copy Explained: How mmap, sendfile, and DMA Boost RocketMQ & Kafka

Traditional I/O

Conventional I/O relies on the read() system call to move data from disk to the kernel buffer and then to user space, followed by write() to copy data from user space to the socket buffer and finally to the network card. This process incurs four user‑kernel context switches and four memory copies.

The sequence includes:

User process calls read(), switching from user mode to kernel mode.

DMA controller copies data from disk to the read buffer.

CPU copies data from the read buffer to the application buffer; read() returns.

User process calls write(), switching back to kernel mode.

CPU copies data from the application buffer to the socket buffer.

DMA controller moves data from the socket buffer to the NIC; write() returns.

Zero‑Copy Concept

Zero‑copy means the CPU does not need to copy data between separate memory regions, reducing both CPU cycles and memory‑bandwidth usage during network transmission.

In practice, zero‑copy still involves some copying, but it eliminates many user‑kernel transitions and CPU‑side copies.

mmap + write

The mmap() system call maps a file directly into the process address space, replacing the read() step and saving one CPU copy. The kernel and user buffers share the same memory region.

The flow involves four context switches and three copies:

User calls mmap(), entering kernel mode.

DMA moves data from disk to the read buffer.

Kernel returns to user mode after mmap() completes.

User calls write(), switching to kernel mode.

CPU copies data from the read buffer to the socket buffer.

DMA transfers data from the socket buffer to the NIC; write() returns.

Using mmap saves one CPU copy and reduces memory usage, making it suitable for large‑file transfers.

sendfile

The sendfile() system call, introduced after Linux 2.1, transfers data directly within kernel space, eliminating the user‑space copy and reducing two context switches compared with read()+write().

The process includes two context switches and three copies:

User calls sendfile(), entering kernel mode.

DMA copies data from disk to the read buffer.

CPU moves data from the read buffer to the socket buffer.

DMA transfers data from the socket buffer to the NIC; sendfile() returns.

Because the transferred data never reaches user space, sendfile is ideal for static file servers.

sendfile + DMA Scatter/Gather

Since Linux 2.4, sendfile can use DMA scatter/gather, allowing the kernel to describe the data layout and let DMA move it directly to the NIC, eliminating CPU copies entirely.

This variant performs only two context switches and two copies, with zero CPU copying, delivering the highest possible I/O throughput on hardware that supports scatter/gather.

User calls sendfile(), switching to kernel mode.

DMA scatter copies data from disk into fragmented kernel buffers.

CPU writes file descriptors and lengths into the socket buffer.

DMA gather moves the described data from kernel buffers to the NIC. sendfile() returns, switching back to user mode.

Application Scenarios

Both RocketMQ and Kafka employ zero‑copy. RocketMQ uses mmap+write for persisting messages, while Kafka combines mmap+write for persistence and sendfile for delivering messages to consumers.

Summary

CPU‑IO speed gaps led to DMA, which offloads data movement from the CPU. Traditional read+write incurs two DMA copies, two CPU copies, and four context switches. mmap+write reduces this to two DMA copies, one CPU copy, and four switches, saving memory for large files. sendfile further cuts context switches to two while keeping one CPU copy, making it ideal for static file serving. sendfile+DMA gather eliminates CPU copies altogether, achieving the best performance on supported hardware.