RocketMQ Secrets: Sequential Writes, Page Cache, Async Flush, Zero‑Copy

Disk Sequential Write

RocketMQ writes messages sequentially to disk, which greatly reduces disk seek time and improves write efficiency. Messages are stored in a CommitLog, a sequential log file; new messages are appended to the end, ensuring ordered writes and high performance.

Page Cache Technology

RocketMQ leverages the operating system's page cache to accelerate writes. Data is first written to a MappedByteBuffer, which maps to a memory region backed by the file, causing the data to reside in the OS page cache before being flushed to disk.

byte[] messageData = "example message".getBytes();<br/>mappedByteBuffer.put(messageData);

The system periodically flushes the page cache to disk, using an asynchronous flush strategy that writes data to the cache first and lets the OS persist it at an appropriate time.

Asynchronous Flushing

Asynchronous flushing (ASYNC_FLUSH) returns immediately after a message is written, boosting throughput by avoiding the latency of synchronous disk writes. While this improves performance, it introduces a risk of data loss if the system crashes before the cache is persisted.

Zero‑Copy

Zero‑copy reduces the number of memory copies during data transmission and storage, enhancing speed. Traditional data transfer involves copying from user space to kernel space, then to disk or network buffers, and back again. Zero‑copy eliminates these extra copies.

On Linux, the sendfile system call can transfer file data directly from a file descriptor to a network socket, bypassing user‑space buffers.

ssize_t sent_bytes = sendfile(socket_fd, file_fd, NULL, file_size);

These techniques collectively enable RocketMQ to achieve high performance and low latency in large‑scale messaging systems.