How RocketMQ’s Storage Model Powers Fast Message Retrieval

1 Overview

RocketMQ’s architecture consists of four roles: Producer, Consumer, NameServer, and BrokerServer. The focus of this article is the BrokerServer’s message storage model.

Data File (CommitLog)

All messages are written sequentially to the commitlog file. When a file reaches its default size of 1 GB, a new file is created with a name based on the starting offset (e.g., 00000000000000000000 for the first file). Sequential writes improve disk I/O performance, enable fast binary search by physical offset, and allow locating a message via its offsetMsgId, which combines the broker’s IP/port and the commitlog offset.

Consume Queue (ConsumeQueue)

The consumequeue stores consumption queues per topic and queue, supporting the publish‑subscribe model. Each queue directory contains consumequeue files written sequentially, with each entry (20 bytes) mapping a logical offset to the physical offset in the commitlog. This design lets consumers retrieve messages without scanning the entire commitlog.

Index File (IndexFile)

Each message’s key is indexed in a hash‑based index file, enabling fast lookup by topic or key. The index file consists of a header, a slot table (default 5 million slots), and an index linked list (up to 20 million entries). The file size is about 400 MB and is named with a timestamp.

<code>//订单Id
String orderId = "1234567890";
message.setKeys(orderId);
</code>

Conclusion

RocketMQ’s storage model perfectly fits the publish‑subscribe paradigm, separates responsibilities among data, consume, and index files, and supports efficient queries by key, offsetMsgId, or tag, making it highly scalable and reliable.