How RocketMQ Stores Messages: Inside the Broker’s CommitLog, ConsumeQueue, and IndexFile

1 Introduction

After covering message sending and communication, the key issue is ensuring message stability after the broker receives a message, preventing loss and guaranteeing durability through persistent storage.

2 Broker Storage Architecture

RocketMQ uses a file‑based storage mechanism (similar to Kafka), storing messages directly on disk rather than using external systems like Redis or MySQL. The storage files reside under ROCKETMQ_HOME and consist of three components.

2.1 CommitLog Message Metadata

All messages are sequentially written to CommitLog files, each fixed at 1 GB. CommitLog has the following characteristics:

Default file size is 1 GB.

File names are 20 characters long, representing the offset and padded with zeros when necessary.

When a 1 GB file is full, writing continues in the next file, maintaining sequential order for high write efficiency.

2.2 ConsumeQueue Message Logical Queue

ConsumeQueue stores indexes of messages in CommitLog; each MessageQueue has its own file, recording which consumer groups have consumed which CommitLog entries. Its features include:

Each entry is 20 bytes: 8‑byte physical offset, 4‑byte message length, and 8‑byte tag hashcode.

Only offset information is stored, making the structure lightweight and fast to load into memory.

Consistency is ensured because CommitLog contains all information (ConsumeQueue, Message Key, Tag, etc.); if a ConsumeQueue is lost, data can be recovered.

Storage path follows HOME/store/consumequeue/{topic}/{queueId}/{fileName} .

2.3 IndexFile Index File

IndexFile is an optional hash‑based index that enables message lookup by key or time range without affecting the main send/consume flow. Its characteristics are:

Implemented as a hash index similar to Java's HashMap, using key hashcode modulo the number of slots and chaining to resolve collisions.

Named by creation timestamp; each file is about 400 MB and can store roughly 20 million indexes.

When a message is sent, the following steps occur:

The message is formatted into the CommitLog structure and written sequentially to a CommitLog file.

The broker creates an index record in the corresponding ConsumeQueue.

An IndexFile is created as needed.

3 Storage Execution Process

Both Kafka and RocketMQ support throughput exceeding 100 k messages per second; the storage architecture enables this performance.

When the broker starts, NettyRemotingServer begins listening on ports for client connections.

Upon a client request, NettyRemotingServer's WorkerGroup handles the readable event and invokes processRequestCommand to process the incoming data.

The broker validates the message via DefaultMessageStore , checking broker response, role (master/slave), write permissions, topic length (≤256 characters), message length (≤65536 characters), and PageCache load; any violation results in rejection.

DefaultMessageStore calls CommitLog.putMessage to store the message. It selects an appropriate CommitLog file for writing. Each CommitLog (1 GB) maps to a MappedFile ; multiple MappedFiles form a MappedFileQueue . MappedFile holds a file channel but writes through a MappedByteBuffer to improve efficiency. Finally, the data is flushed to the physical CommitLog file.

4 Summary

Understand the three core components of RocketMQ broker storage: CommitLog, ConsumeQueue, and IndexFile.

When the broker receives a storage request, it writes the message to a MappedFile, which maps to a CommitLog file via a memory‑mapped buffer.