How RocketMQ’s IndexFile Enables Near‑O(1) Message Lookups

1. IndexFile Physical Structure

Index files are stored in $HOME/store/index/, named by creation timestamp (e.g., 20240916081000), with a default size of 400 MB. Each file consists of three parts: a 40‑byte header, a hash slot area, and an index item area.

Header (40 Bytes)

beginTimestamp – earliest message storage time in the file

endTimestamp – latest message storage time in the file

beginPhyoffset – physical offset of the earliest message in the CommitLog

endPhyoffset – physical offset of the latest message in the CommitLog

hashSlotCount – number of hash slots already used

indexCount – number of index entries written

Hash Slot Area

Acts like a hash‑table bucket array with a default of 5 million slots, each 4 bytes.

Each slot stores the head index position of a linked list, not the message data itself.

Index Item Area

Each index entry occupies 20 bytes and can store up to 20 million entries.

Fields per entry:

keyHash – hash value of the message key

phyOffset – physical offset of the message in the CommitLog

timeDiff – difference between the message store time and the header's beginTimestamp

prevIndex – pointer to the previous index with the same hash (linked‑list chain)

Logical chain: Hash Slot → Index Item (linked list) → CommitLog offset .

2. Index Construction (Write Path)

After a message is successfully appended to the CommitLog, if it carries a Key (or UNIQ_KEY), the ReputMessageService builds the index asynchronously.

Compute hash: calculate keyHash from the Key.

Locate slot: slotPos = keyHash % slotNum.

Handle collision:

Read the current slot value ( currentSlotValue).

Set prevIndex = currentSlotValue for the new Index Item.

Update data:

Write the new index as the head of slotPos.

Write the Index Item fields ( phyOffset, keyHash, timeDiff).

Update Header: refresh endTimestamp, endPhyoffset, and indexCount.

Index building is a sequential write plus linked‑list chaining operation, avoiding random I/O.

3. Index Query (Read Path)

Key‑based lookup proceeds as follows:

Select candidate IndexFiles by filtering timestamps using the Header's beginTimestamp and endTimestamp.

Compute keyHash and locate the slot position slotPos.

Traverse the linked list:

Read the slot head index position ( indexPos).

Iterate Index Items, comparing keyHash. When a match is found, verify the real key via phyOffset to handle possible hash collisions.

If not matched, follow prevIndex to the previous entry until the chain ends.

Collect all matching phyOffset values and fetch the corresponding messages from the CommitLog.

Because the hash chain is typically very short, query complexity is close to O(1).

4. Performance Advantages

Sequential writes align with CommitLog, eliminating random I/O.

Index files are accessed via memory‑mapped (MMAP) I/O, giving read/write speeds near memory latency.

Time‑partitioned IndexFiles allow rapid exclusion of irrelevant files.

Fixed 400 MB file size with pre‑allocation avoids expansion overhead.

Asynchronous index construction does not block the main message‑write flow.

5. Limitations

Supports only exact key/UNIQ_KEY/timestamp queries; fuzzy, range, or complex conditions are not available.

A message must contain a Key to be indexed.

Hash collisions are rare but require final verification of the real key.

Index files are not permanent; they expire (default 3 days) together with the CommitLog and are deleted when the associated CommitLog file is removed.

Heavy keyed traffic can cause index files to grow to tens of gigabytes, requiring careful disk planning.

6. Relationship with CommitLog and ConsumeQueue

CommitLog – primary sequential storage of all message payloads.

ConsumeQueue – stores phyOffset, msgSize, and tagHashCode for ordered consumption.

IndexFile – provides fast key/timestamp lookup.

These three components complement each other: CommitLog is the data source, ConsumeQueue enables sequential consumption, and IndexFile enables precise message retrieval.

7. IndexFile Lifecycle

Default expiration policy: 3 days, aligned with CommitLog.

Deletion condition: when the corresponding CommitLog file is removed, its IndexFile is also deleted.

Disk planning: with high message volume and frequent keyed messages, index files can reach dozens of GB.

8. Bottlenecks & Optimization Tips

Hot keys (very high frequency) can lengthen hash chains; avoid overusing a single key at the business layer.

Account for the additional storage overhead of index files when provisioning disk space.

For complex queries (fuzzy, range), integrate external search systems such as Elasticsearch.

9. Practical Use Cases

Message tracing: use UNIQ_KEY to quickly locate a message for loss investigation.

Business queries: e.g., locate messages by order number as the key.

Time‑window queries: fetch all messages within a specific time range.

10. Comparison with Other MQs

Kafka relies on partition + offset and cannot perform key‑based queries.

RabbitMQ uses routing keys but lacks file‑level hash indexes.

RocketMQ’s file‑based hash table index offers the strongest query capability among open‑source message queues.

Conclusion

RocketMQ’s index mechanism implements a file‑level hash table with linked‑list collision resolution, leveraging sequential writes, MMAP, and time partitioning to achieve high‑performance, near‑O(1) message location. Understanding this design is essential for operations, troubleshooting, and effective system architecture.