Unlocking RocketMQ 4.x: Deep Dive into Architecture, Protocols, and High‑Performance Messaging

1 Overall Architecture

RocketMQ 4.x consists of four roles: NameServer, BrokerServer, Producer, Consumer.

NameServer is a stateless node that registers and discovers topics, similar to Zookeeper.

BrokerServer stores, delivers and queries messages, ensuring high availability.

Producer selects a broker queue via load balancing and sends messages with fast‑fail and low latency.

Consumer receives messages in push or pull mode.

Cluster workflow: start NameServer, start Broker (register with NameServer), create Topic, Producer sends messages, Consumer pulls messages.

2 Publish‑Subscribe Model

Traditional ActiveMQ uses point‑to‑point; RocketMQ and Kafka use publish‑subscribe where producers publish to topics and multiple subscribers receive the same message.

3 Communication Framework

01 Communication Protocol

Message consists of four parts: total length (int), serialization type & header length (int), serialized header data, and body bytes (default JSON).

02 Reactor Model

Reactor abstracts three components: Reactor (event dispatcher), Acceptor (connection handler), Handlers (business logic). RocketMQ uses a master‑worker thread model with independent business thread pools per request type.

Reactor boss thread ( eventLoopGroupBoss) listens for TCP connections, registers channels; worker threads handle I/O, SSL, codec, idle checks, then business logic via processor table.

4 File Storage Mechanism

Broker stores messages in three files: commitlog (message body and metadata), consumequeue (index for fast consumption), and indexfile (key or time‑range lookup).

All queues share a single commitlog; messages are persisted synchronously or asynchronously, and background threads build consumequeue and indexfile.

5 High‑Performance Read/Write

01 Sequential Write

Messages are appended sequentially to commitlog, reducing random I/O and enabling binary search by physical offset.

02 Memory‑Mapped I/O

RocketMQ uses mmap + write zero‑copy, involving one CPU copy and two DMA copies, which is faster than traditional read/write.

6 Consumption Process

Consumer obtains load‑balanced queues, creates a pullRequest with a processQueue (TreeMap), pulls messages asynchronously, submits them to a thread pool, invokes the listener, updates offsets in memory and periodically reports to the broker.

7 Traditional Deployment Modes

01 Dual‑Master

All nodes are masters; simple configuration, high performance, but a single machine failure delays its own messages.

02 Multi‑Master Async

Each master has multiple slaves with asynchronous replication; fast, but a few messages may be lost on failure.

03 Multi‑Master Sync

Synchronous replication guarantees no loss but adds ~10% latency and no automatic master switch.

8 DLedger Cluster Deployment

DLedger uses Raft for automatic leader election without external services. A DLedger group requires at least three nodes; the leader replicates data to followers, providing high availability and horizontal scaling.

9 Transaction Messages

Producer sends a half‑transaction message, executes local transaction, then commits or rolls back. If the broker does not receive the final status, it performs a message check and retries.

10 Broadcast Messages

In broadcast mode each message is delivered to all consumers, useful for push notifications and cache synchronization.

11 Ordered Messages

Two types: partition‑ordered (messages with same sharding key go to the same queue) and global‑ordered (single‑queue topic). Producers select queues via a hash selector; consumers must consume each queue with a single thread and acquire broker locks.

12 Architecture Drawbacks

Master‑Slave lacks failover; DLedger solves master failure but requires ≥3 replicas, incurs Raft quorum latency, and needs duplicated storage logic. Client‑side logic is heavy and idempotence is critical.

13 RocketMQ 5.0 New Mode

Introduces a stateless proxy layer separating client protocol, permission, and consumption management from the broker, enabling elastic scaling in cloud environments while remaining compatible with the 4.x architecture.