Unlocking RocketMQ: Core Features, Architecture, and Performance Secrets

RocketMQ is a distributed queue-model middleware with features: strict message ordering, rich pull modes, efficient subscriber horizontal scaling, real-time subscription, and ability to accumulate billions of messages.

RocketMQ Network Deployment Characteristics

(1) NameServer is a near-stateless node that can be clustered without any information synchronization between nodes.

(2) Broker deployment is more complex: each Broker has a Master and Slaves. A Master can have multiple Slaves, but a Slave corresponds to only one Master. The relationship is defined by the same BrokerName and different BrokerId (0 for Master, non‑zero for Slave). Each Broker maintains long connections with all NameServer nodes and periodically registers Topic information.

(3) Producer establishes a long connection with a randomly selected NameServer node, periodically obtains Topic routing information, then connects to the Master providing the Topic service and sends heartbeats. Producers are stateless and can be clustered.

(4) Consumer also connects to a randomly selected NameServer node, obtains routing information, and establishes long connections with the Master and Slaves, sending heartbeats. Consumers can subscribe from either Master or Slave according to Broker configuration.

RocketMQ Storage Characteristics

(1) Zero‑copy principle: Consumer uses zero‑copy during message consumption. RocketMQ employs the mmap+write method, which is more efficient for small block transfers than sendfile.

Advantages: high efficiency even with frequent small‑file transfers.

Disadvantages: cannot fully leverage DMA, higher CPU usage, complex memory safety, and risk of JVM crashes.

b) Sendfile method

Advantages: utilizes DMA, lower CPU consumption, high efficiency for large file transfers, no new memory safety issues.

Disadvantages: lower efficiency for small blocks, limited to BIO transfer, cannot use NIO.

(2) Data storage structure

Key Features of RocketMQ

Single Machine Supports Over 10,000 Persistent Queues

(1) All data is stored separately in a Commit Log, written sequentially and read randomly.

(2) The queue presented to end users stores only the position of the message in the Commit Log and flushes to disk serially.

Benefits:

Lightweight queues with minimal per‑queue data.

Serial disk access avoids contention, preventing IOWait increase as queues grow.

Drawbacks:

Writes are sequential but reads become random.

Reading a message requires first reading the Consume Queue then the Commit Log, adding overhead.

Ensuring full consistency between Commit Log and Consume Queue adds programming complexity.

Mitigations:

Random reads aim to hit page cache; larger memory reduces disk access impact.

Page cache prefetching and NOOP I/O scheduler can turn random reads into more efficient sequential jumps, improving performance up to 5×.

Consume Queue is tiny and read sequentially, so its performance remains near memory speed even under load.

Commit Log contains all metadata (similar to MySQL/Oracle redo log); even if Consume Queue is lost, data can be recovered from the Commit Log.

Flushing Strategies

All messages are persisted: first written to the system page cache, then flushed to disk, ensuring both memory and disk hold a copy and allowing direct memory reads.

Asynchronous flushing:

Disk speed often exceeds network speed, so the system can return to the client after writing to memory while a background thread flushes to disk.

If memory pressure arises, clean pages are discarded using LRU; if insufficient clean pages remain, the write blocks and the system flushes about 32 pages to free space, preventing memory overflow.

Synchronous flushing:

After writing to page cache, the thread waits for the flush thread to complete, then the waiting thread is awakened and returns success to the client.

Message Query

3.1 Query by MessageId

MessageId is 16 bytes, containing the broker address and Commit Log offset. The broker address and offset are extracted from the MessageId to locate and decode the full message.

3.2 Query by Message Key

1. Compute hashcode of the key and modulo slotNum to locate the slot (slotNum is the maximum number of slots in the index file, e.g., 5,000,000).

2. Use slotValue to find the last index entry in the slot’s list (slotValue always points to the newest entry).

3. Traverse the index list to return results within the query time range (default maximum 32 records per request).

4. Handle hash collisions by comparing stored key hashcodes and, if necessary, the actual key string on the client side.

5. Index entries store time as a delta from the file’s start time; the index file has a fixed length and structure.

Server‑Side Message Filtering

RocketMQ filters messages at subscription time rather than during production.

1. On the broker, Message Tags are stored as hashcodes in the Consume Queue. The broker compares the stored hashcode with the subscription tag’s hashcode; mismatches are skipped.

2. After receiving filtered messages, the consumer re‑checks the actual tag string to ensure correctness.

Reasons:

Storing tags as hashcodes saves space in the fixed‑length Consume Queue.

Filtering does not access the Commit Log, allowing efficient operation even under heavy backlog.

Any hash collisions are resolved on the consumer side, guaranteeing accuracy.

Utilizing Large JVM Memory

1. Producer sends messages, which enter the Java heap via socket.

2. Messages move from the Java heap to the page cache (physical memory).

3. An asynchronous thread flushes messages from page cache to disk.

4. In normal consumption, messages are transferred directly from page cache to socket, bypassing the Java heap; with 96 GB of memory, about 100 million 1 KB messages can be cached.

5‑8. In abnormal consumption, page cache misses trigger disk I/O to load messages before sending.

Message Backlog Solutions

1. Backlog capacity depends on disk size.

2. Throughput impact varies: without slaves, some impact; with slaves, minimal impact.

3. Consumer impact follows the same pattern.

4. Disk‑based backlog access throughput drops to around 5,000 TPS, related to concurrency.

When slaves are present, the master redirects consumers to pull from slaves, isolating backlog handling from normal traffic. Slave writes prioritize throughput over latency, using batch sequential writes (e.g., 1 MB batches), so write performance remains high despite longer write latency.

Source: http://www.uml.org.cn/zjjs/201504011.asp