Unlocking RocketMQ 4.x: Architecture, Protocols, and High‑Performance Messaging

This article takes you deep into the core knowledge system of RocketMQ 4.x, from architecture design to key mechanisms, exploring the underlying logic of this highly available messaging middleware.

Overall Architecture

RocketMQ 4.x architecture includes four roles:

NameServer : a nearly stateless node that can be clustered, acting as a simple Topic routing registry similar to Zookeeper in Dubbo, supporting dynamic registration and discovery of Brokers.

BrokerServer : responsible for message storage, delivery, query, and high‑availability guarantees.

Producer : publishes messages, selects appropriate Broker queues via load‑balancing, and supports fast failure with low latency.

Consumer : consumes messages, supporting both push and pull modes.

RocketMQ cluster workflow:

Start NameServer, which listens for connections from Brokers, Producers, and Consumers.

Broker starts, maintains long connections with all NameServers, sends heartbeat packets containing its IP, port, and Topic information; successful registration creates a mapping between Topics and Brokers in the NameServer cluster.

Before sending or receiving messages, create a Topic, specifying which Brokers store it; topics can also be auto‑created on message send.

Producer connects to a NameServer, obtains the list of queues for the target Topic, selects a queue, establishes a long connection with the corresponding Broker, and sends the message.

Consumer connects to a NameServer, obtains the list of Brokers for its subscribed Topic, connects directly to the Broker, and consumes messages.

Publish‑Subscribe Model

Traditional ActiveMQ uses a point‑to‑point model where a producer sends to a specific queue and a consumer receives from that queue, ensuring each message is processed by only one consumer.

RocketMQ and Kafka adopt a publish‑subscribe model: producers publish to a topic, and multiple subscribers receive the same message, suitable for one‑to‑many or many‑to‑many communication such as real‑time event processing, log handling, and notification systems.

Communication Framework

01 Communication Protocol

The transmission content consists of four parts:

Message length : total length stored in four bytes (int).

Serialization type & header length : one byte for serialization type, three bytes for header length.

Message header data : serialized header data.

Message body data : binary content of the message body.

By default, the header data is serialized in JSON format.

02 Reactor Model

The Reactor thread model abstracts three components:

Reactor : listens and dispatches events, acting as the scheduling center.

Acceptor : handles I/O connection requests.

Handlers : execute the business logic for each event.

Remoting uses a typical master‑worker multithreaded model with independent business thread pools for different request types.

A Reactor boss thread ( eventLoopGroupBoss) listens for TCP connections, creates SocketChannel, and registers it to a selector. RocketMQ automatically selects NIO or Epoll based on the OS, then listens for actual network data.

Received data is handed to a worker thread pool ( eventLoopGroupSelector) for SSL verification, codec, idle check, and connection management, delegated to defaultEventExecutorGroup. Business logic is processed by a business thread pool selected via the request code from processorTable, wrapped as a task and submitted to the appropriate executor.

File Storage Mechanism

RocketMQ stores messages in three tightly related files within the broker's storage directory:

CommitLog : the main storage for message bodies and metadata.

ConsumeQueue : a consumption queue that improves consumption performance.

IndexFile : provides key or time‑range based message lookup.

All queues share a single CommitLog. Producers send messages to the broker, which persist them to the CommitLog synchronously or asynchronously. Once persisted, messages are not lost. Background threads asynchronously build ConsumeQueue and IndexFile.

High‑Performance Read/Write

01 Sequential Write

Messages are written sequentially to files, which reduces random I/O and leverages faster disk sequential access. Sequential writes can achieve hundreds of MB/s, while random writes drop to a few hundred KB/s.

Each message’s physical offset is unique; the CommitLog file name is incremental, allowing binary search to locate a message by its offset.

02 Memory‑Mapping Mechanism

Linux’s mmap maps kernel buffers to user space, enabling zero‑copy transfers. The mmap + write approach reduces CPU copies and context switches compared to traditional read + write.

The data flow involves:

User process calls mmap(), switching to kernel mode.

Kernel maps the read buffer to user space.

CPU uses DMA to copy data from memory or disk to the kernel read buffer.

Kernel returns to user mode.

User process calls write(), switching to kernel mode.

CPU copies data from the read buffer to the network buffer.

DMA transfers data from the network buffer to the NIC.

Kernel returns to user mode.

RocketMQ adopts the mmap + write zero‑copy method, suitable for small‑block message persistence, whereas Kafka uses sendfile for large‑block log data.

Consumption Process

Consumer starts and triggers the load‑balancing service, which assigns queues to the consumer instance.

The service creates a pullRequest for each assigned queue; the request holds a processQueue (a red‑black tree TreeMap) that stores pulled messages.

A single‑threaded pull service pops pullRequest from pullRequestQueue, performs asynchronous pull, and places messages into the processing queue.

After a pull completes, the request is reused and re‑queued into pullRequestQueue.

The pull service invokes consumeMessageService.submitConsumeRequest, which uses a consumer thread pool.

Consumer threads invoke listener.consumeMessage to process messages.

On successful consumption, the offset is updated in memory ( offsetTable) and periodically reported to the broker; on failure, the message is sent back to the broker.

The broker updates consumption progress via commitOffset and periodically flushes it to consumerOffset.json.

Traditional Deployment Modes

01 Dual‑Master Mode

All nodes are masters (e.g., 2 or 3 masters) with no slaves.

Advantages : simple configuration, high performance; a single master failure does not affect applications; with RAID10, messages are not lost.

Disadvantages : if a machine goes down, its unconsumed messages are delayed until recovery, affecting real‑time consumption.

02 Multi‑Master Multi‑Slave (Async)

Each master has multiple slaves; HA uses asynchronous replication with millisecond‑level delay.

Advantages : minimal message loss on disk failure; consumers can read from slaves when a master fails, transparent to applications.

Disadvantages : a small amount of message loss may occur during master failure or disk damage.

03 Multi‑Master Multi‑Slave (Sync)

Each master has multiple slaves; HA uses synchronous double‑write, requiring acknowledgment from both master and slaves before responding to the client.

Advantages : no single point of failure; high data and service availability even when a master is down.

Disadvantages : slightly lower performance (≈10% slower) and higher latency; slaves do not automatically become masters on failure.

Deleger Cluster Deployment

Before RocketMQ 4.5, only Master/Slave deployment existed, which required manual failover. The new DLedger mode uses Raft for automatic leader election, eliminating external coordination services.

Two DLedger groups (e.g., RaftNode00 and RaftNode01) are defined; each group requires at least three machines (odd number) with a broker on each. Raft automatically elects a leader, and followers replicate data, providing high availability and horizontal scalability.

Transaction Messages

RocketMQ transaction messages ensure eventual consistency between message production and local transactions.

Producer sends a message to the broker.

Broker persists the message and returns an ACK, marking the message as “pending delivery” (half‑transaction message).

Producer executes local transaction logic.

Producer sends a second confirmation (Commit or Rollback) to the broker. On Commit, the broker marks the message as deliverable; on Rollback, the message is discarded.

If the broker does not receive a confirmation (e.g., network failure), it performs a message check after a timeout, prompting the producer to resend the final status.

Broadcast Messages

In broadcast consumption mode, each message is pushed to all consumers in the cluster, ensuring at least one consumption per consumer. Typical scenarios include message push (e.g., driver dispatch notifications) and cache synchronization across distributed nodes.

Ordered Messages

Ordered messages guarantee FIFO order for a specific Topic. There are two types:

Partitioned ordered messages : messages with the same Sharding Key are routed to the same queue, preserving order within that partition.

Global ordered messages : the Topic has only one partition, so all messages follow a strict global FIFO order.

Producers implement MessageQueueSelector to hash the Sharding Key and select the target queue. Consumers must consume each queue with a single thread to maintain order. The ordered consumption service ( ConsumeMessageOrderlyService) acquires a lock from the broker for the assigned queue; if the lock acquisition fails, it retries after a delay.

Architecture Drawbacks

RocketMQ has two deployment architectures: Master‑Slave and Deleger.

Master‑Slave issues:

When a master fails, message sending to that group stops, reducing writable partitions and affecting order‑sensitive workloads.

Operations that rely on the master (e.g., offset locking, max/min offset queries) become unavailable, impacting ordered consumption and control operations.

Secondary message consumption (e.g., delayed or transaction messages) halts because scanning and re‑delivery threads stop, leading to delays or loss.

Deleger (Raft) issues:

Requires at least three replicas per broker group, increasing cost.

Raft majority constraints add latency; a three‑replica group needs two acknowledgments, a five‑replica group needs three, limiting flexibility.

DLedger replaces RocketMQ’s native storage components, preventing reuse of features like TransientPool and zero‑copy without additional porting effort, raising maintenance overhead.