Understanding RocketMQ: Architecture, Modules, and Deployment Essentials

RocketMQ Background

In 2010 Alibaba needed a messaging system that could handle ordered messages and massive accumulation. Meta Q 1.0 was created in 2011, evolved to Meta Q 3.0 in 2012 and abstracted a generic engine called RocketMQ, which was later open‑sourced. By the 2016 Double‑11 shopping festival RocketMQ processed trillion‑level messages and entered Apache incubation.

What Is RocketMQ

A queue‑model middleware offering high performance, high reliability, real‑time processing and native distributed deployment.

Producers, Consumers and Queues can all be deployed across multiple nodes.

Producers publish to a Topic that consists of multiple queues; Consumers can either broadcast (receive all queues of a Topic) or cluster (each instance consumes a subset of queues).

Strict message ordering is guaranteed.

Rich pull‑based consumption modes.

Horizontal scaling for subscribers.

Billions of messages can be accumulated with minimal external dependencies.

Key Terminology

Topic : Logical name that groups a class of messages. Both producers and consumers must specify a Topic.

Tag : Optional label attached to a message; consumers can filter by Tag within the same Topic.

Message Queue (Q) : A Topic is split into multiple Queues. When a Topic is created on different Brokers, each Broker holds its own set of Queues, enabling horizontal scalability.

Offset : The index of a message inside a logical Queue, analogous to an ever‑growing array.

Max Offset : The next offset value (latest offset + 1) indicating where the next message will be stored.

Min Offset : The smallest existing offset; messages with offsets lower than Min Offset have been physically deleted from the Broker.

Consumer Offset : The position from which the next pull request will start.

Core Components

NameServer

Stores the mapping between Topics and Brokers. It is a lightweight, highly stable routing center. Multiple NameServers do not communicate with each other; a single failure does not affect the rest of the cluster. Existing Producers, Consumers and Brokers continue to operate, but new clients cannot join until a NameServer is reachable.

Broker

The core storage module. It persists Topic messages, manages consumer offsets, and enforces a master‑slave model: only the Master accepts writes, while Slaves continuously sync from the Master.

Producer

Stateless client that sends messages. A Producer must specify the target Topic and connects to a NameServer to discover the Brokers that host the Topic. By default it round‑robin selects a Message Queue for each message, distributing load across Queues (and thus across Brokers). Custom queue selection (e.g., hash‑based ordering) can be implemented via the MessageQueueSelector interface.

Consumer

Stateless client that receives messages. Two implementations exist: Push (primary) and Pull (auxiliary). A Consumer connects to a NameServer, fetches the latest Queue list for its subscribed Topics, and then pulls messages directly from the corresponding Brokers.

Cluster Deployment Architecture

Cluster Workflow

NameServer starts, listens on a port and waits for Brokers, Producers and Consumers to register.

Each Broker establishes a long‑lived connection to all NameServers and sends a heartbeat every 30 seconds containing its IP, port and the full list of Topics it manages. Successful registration creates the Topic‑Broker mapping in the NameServer cluster.

Before messaging, a Topic must be created (or auto‑created on first publish) and associated with one or more Brokers.

When a Producer sends a message, it first queries a NameServer for the list of Brokers that host the target Topic, then opens a direct TCP connection to the chosen Broker and publishes the message.

A Consumer follows the same discovery process, then pulls messages from the appropriate Brokers.

Component Feature Details

NameServer

Stores Topic‑Broker relationship data; extremely stable. Multiple NameServers operate independently, so a single failure does not affect the cluster.

Load is dominated by heartbeat processing and Topic‑Broker lookups. If a Broker manages tens of thousands of Topics, a single heartbeat can be dozens of megabytes, which may cause transmission failures on poor networks and lead to false timeout detection.

Broker

High‑Concurrency Read/Write

Sequential write: All Topics share a single log file; when the file reaches 1 GB a new file is created, guaranteeing true sequential disk writes and high TPS.

Random read: RocketMQ relies on the OS page cache. Even a 1 KB read can trigger prefetch of larger blocks, improving cache hit rates and reducing I/O latency.

Load Balancing & Dynamic Scaling

Topic dimension: Increase the number of Queues for a high‑traffic Topic; queue count scales proportionally with send/consume speed.

Broker dimension: Add new Broker nodes when overall cluster load is high. New Brokers register with NameServers, and Producers/Consumers discover them automatically.

High Availability & Reliability

HA: Master‑slave deployment; slaves continuously sync from the master. If the master fails, slaves continue to serve consumption but not writes.

Reliability: Messages are persisted with synchronous or asynchronous flush. Synchronous flush returns success only after data is written to disk; asynchronous flush may lose messages only on a machine crash.

Broker‑NameServer Heartbeat Each Broker sends a heartbeat to all NameServers every 30 seconds, including its current Topic list. If a Broker does not send a heartbeat for two minutes, the NameServer marks it offline and updates the Topic‑Broker mapping. NameServers do not proactively notify Producers or Consumers of Broker failures.

Flush Strategy All messages are first written to the OS page cache and then flushed to disk, ensuring data exists both in memory and on persistent storage. Asynchronous flush : In tests, 15 000 TPS achieved ~300 MB/s sequential write speed; a typical 1 Gbps NIC can handle up to 128 MB/s, so flush throughput can keep up with ingestion. Synchronous flush : The client blocks until the flush completes before receiving a success response.

Producer Details

On startup a Producer specifies the NameServer address, selects one NameServer from the cluster and establishes a long‑lived connection. If the chosen NameServer fails, the Producer automatically reconnects to another.

Heartbeat detection mirrors the Broker mechanism.

Default queue selection is round‑robin across all Queues. Custom selection (e.g., hash of a business key for ordered messages) can be implemented via MessageQueueSelector.

Consumer Details

On startup a Consumer specifies a NameServer address, establishes a long‑lived connection, and every 30 seconds fetches the latest Queue information for all subscribed Topics. If a Broker goes down, the Consumer may take up to 30 seconds to detect the change before reconnecting to remaining Brokers.

Heartbeat detection mirrors the Broker mechanism.

Consumption modes

Cluster consumption : Instances within the same Consumer Group share the load; each message is delivered to only one instance.

Broadcast consumption : Every instance in the Consumer Group receives each message, effectively duplicating delivery.

Load balancing in cluster mode distributes Queues evenly among Consumer instances with the same group ID.

Pull flow: The Consumer periodically sends a pull request to the Broker. If messages are available, the Broker returns them immediately; otherwise the request blocks until data arrives or a timeout occurs. Upon receipt, the Consumer invokes the user‑defined listener callback.

Persisted Commit Log

Message Queues do not store the actual payload. All messages are written sequentially to a shared Commit Log file. Each Queue only keeps pointers (offsets) to locations within the Commit Log, enabling fast lookup while preserving strict ordering across Topics.

Commit Log Cleanup Policy

Physical message files are deleted only when one of the following conditions is met:

The file has expired (default 72 hours) and the scheduled cleanup time (default 04:00) is reached.

The file has expired and disk usage has reached the water‑mark line (default 75%).

Disk usage reaches the mandatory release threshold (default 85%); in this case files are cleaned in bulk regardless of expiration until sufficient space is freed.

When disk usage exceeds the danger line (default 90%), the Broker refuses write requests to protect itself.

References

RocketMQ User Guide v3.2.4

Alibaba Cloud Community

https://yq.aliyun.com/articles/69647?spm=5176.100240.searchblog.7.ZgGuDF

https://yq.aliyun.com/articles/66101?spm=5176.100240.searchblog.101.s7dvlU

https://yq.aliyun.com/articles/66110?spm=5176.100239.blogcont66101.23.kpzm2R

Other Resources

http://blog.csdn.net/damacheng/article/details/42846549

http://jameswxx.iteye.com/blog/2091966