Deep Dive into RocketMQ: History, Architecture, Core Modules, Storage, and Transactional Messaging

1. History of RocketMQ

RocketMQ originated from Alibaba's early messaging engines: the 2007 "Five‑Color Stone" project produced Notify, followed by Napoli in 2010, MetaQ 1.0 in 2011 (a Java rewrite of Kafka), MetaQ 2.0 in 2012, and the open‑source RocketMQ 3.0 released in 2012. In 2015 Aliware MQ and Notify 3.0 were built on top of RocketMQ, and the core engine was donated to Apache in 2016.

2. Principles and Architecture

The cluster consists of NameServer (stateless routing service), Broker‑Master, Broker‑Slave, Producer, and Consumer. Producers obtain routing info from the local cache or NameServer, select a MessageQueue, and send messages to the broker, which stores them on disk.

The communication layer (rocketmq‑remoting) is built on Netty and defines a custom protocol for efficient client‑server interaction.

3. Source Code Directory

Key packages in RocketMQ 4.4.0 (Netty 4.0.42.Final) include:

rocketmq‑broker: core broker handling requests and storage.

rocketmq‑client: Java client implementation.

rocketmq‑common: shared utilities and constants.

rocketmq‑example: sample code for ordered messages, push/pull consumers.

rocketmq‑filter: message filtering service.

rocketmq‑namesrv: name service for broker discovery.

rocketmq‑remoting: Netty‑based communication module.

rocketmq‑srvutil: command‑line utilities.

rocketmq‑store: storage layer with CommitLog, ConsumeQueue, Index files.

rocketmq‑tools: management utilities.

4. Service Startup

The entry point is org.apache.rocketmq.broker.BrokerController, which initializes thread pools for sending, consuming, and cleaning expired requests.

5. Client Message Flow

Producers use MQProducer → sendKernelImpl → MQClientAPIImpl.sendMessage → sendMessageAsync → NettyRemotingClient.invokeAsync to transmit messages.

6. Remoting Layer

NettyRemotingClient and NettyRemotingAbstract handle asynchronous invocation, writing to the channel and awaiting selector events.

7. Broker Processing

Broker receives messages via SendMessageProcessor.processRequest, which calls MessageStore.putMessage. The implementation DefaultMessageStore ultimately writes to CommitLog using MappedFile and AppendMessageCallback.

8. Storage Layer

Messages are stored in three large file types (CommitLog, ConsumeQueue, Index) and several small files. MappedByteBuffer (wrapped as MappedFile) provides high‑performance random read and sequential write. Flushing is performed by FlushRealTimeService, CommitRealTimeService, and CommitLog internal classes, with configurable intervals and page thresholds.

9. High‑Availability (HA) Replication

Two replication modes exist: SYNC_MASTER (synchronous double‑write) and ASYNC_MASTER (asynchronous). Synchronous mode waits for the slave to acknowledge the write, while asynchronous mode returns after the master write, offering higher throughput when strong consistency is not required.

10. Transactional Messages

RocketMQ supports transactional messaging to achieve eventual consistency in distributed transactions. The flow includes sending a prepare message (invisible to consumers), executing a local transaction, and then either committing or rolling back the message. If the broker does not receive a commit/rollback within a timeout, it performs a transaction status check (callback) to the producer.

Transactional message code resides under broker.transaction, and newer versions have removed the built‑in transaction check interface, requiring custom implementation.

Overall, RocketMQ’s design emphasizes modularity, high performance through NIO‑based storage, flexible flushing strategies, and robust HA and transactional capabilities suitable for large‑scale distributed systems.