Deep Dive into Apache RocketMQ: Architecture, Routing, Storage, and High‑Availability Design

How to effectively learn an open‑source project is largely determined by the learning path: fork a demo and test it, read the official documentation, examine the source code, or consult community‑recommended books.

The article is authored by senior Apache RocketMQ user Ding Wei, who co‑authored the book RocketMQ Technical Insider: Architecture Design and Implementation Principles to illustrate the core principles of RocketMQ, including topic routing registration and removal, high‑availability message sending, message storage file design, concurrent pull and consumption, master‑slave synchronization (HA), and transaction messaging.

Topic Routing Mechanism

Before describing the routing registration mechanism, the overall RocketMQ architecture is introduced:

Producer : sends messages to the broker.

NameServer : routing registration center.

Broker : message storage server.

Consumer : message consumer (not shown in the diagram).

Programming techniques involved:

A. Long‑connection model and heartbeat packets.

B. Multithreaded programming with classic read‑write lock usage.

Thought: Since a producer cannot instantly detect a broker failure, how is high‑availability of message sending ensured?

Message Sending High‑Availability Design

The default queue load uses a round‑robin strategy, and a retry mechanism guarantees high availability. When a broker goes down, the producer stops selecting its queues for a configurable period (e.g., five minutes), thereby avoiding the faulty broker while retries ensure delivery.

Message Storage File Design

CommitLog File : a sequential write file (default 1 GB) named by the first global offset, enabling fast location of a message by its physical offset.

ConsumeQueue File : a topic‑based index file where each entry has a fixed length (8 bytes offset, 4 bytes length, 8 bytes tag hash). This fixed‑size design allows direct array‑like access to entries, greatly improving read performance.

IndexFile : a hash‑based index on disk, consisting of a 40‑byte header, 5 million hash slots (4 bytes each), and 20 million index entries (20 bytes each) storing the key hash, physical offset, timestamp, and a pointer to the previous entry.

Programming techniques for storage files:

A. Memory‑mapped file usage.

B. Memory locking.

C. File‑based hash index implementation.

D. Multithreaded coordination.

E. Asynchronous flush mechanisms.

Concurrent Message Pull and Consumption Process

Consumption involves queue load balancing, message pulling, filtering, processing, and progress feedback. The RebalanceService thread redistributes queues every 20 seconds; new queues create PullRequest objects that are added to the PullMessageService queue, while removed queues are marked as discarded.

Consumers report progress after each message, using the smallest offset in the processing queue to avoid message loss, though this may cause duplicate consumption.

To prevent memory overflow when the processing queue grows, RocketMQ applies flow control: if the number of pending messages exceeds 1 000 or the total size exceeds 100 MB, pulling is temporarily paused and delayed.

Master‑Slave Synchronization (HA)

The HA process involves the master listening on a port, the slave establishing a TCP connection, periodic pull requests (every 5 seconds), writing received data to the local commitlog, reporting progress to the master, and repeating the cycle.

Transaction Messages

Transaction messaging follows a two‑phase commit model. The producer sends a Prepare message, records the transaction state locally, and the broker stores the message under a special system topic. A dedicated thread periodically checks the transaction status; if the local transaction succeeded, the broker commits the message, otherwise it rolls back. After five consecutive unknown checks, the broker rolls back by default.

Learning Recommendations

A. Read the official RocketMQ documentation for a global view.

B. Set up a local debugging environment (NameServer, Broker) and run the example code in the example package.

C. Study functional modules step‑by‑step (sending, storage, consumption, HA, etc.) without jumping between topics.

Author: Ding Wei, RocketMQ official live‑stream instructor and co‑author of RocketMQ Technical Insider .