Unveiling the Full Lifecycle of a RocketMQ Message: From Production to Deletion

Core Concepts

NameServer : a registration center that stores topic routing information and manages brokers; NameServers do not communicate with each other.

Broker : the core component that persists messages; multiple broker instances with the same name form a broker group.

Topic : the logical name of a collection of messages; a topic can be spread across different broker groups.

Queue : each topic contains multiple queues (default 4 per broker group); the total number of queues grows with the number of broker groups.

Producer : the entity that creates messages.

Producer Group : a logical grouping of producers; all producers in the same group share configuration.

Consumer : the entity that reads messages.

Consumer Group : a logical grouping of consumers; consumers in the same group consume independently from other groups.

Message Birth and Sending

Producers instantiate a DefaultMQProducer, set the NameServer address, start the producer, build a Message, and call producer.send(msg) to deliver the message.

DefaultMQProducer producer = new DefaultMQProducer("sanyouProducer");
producer.setNamesrvAddr("localhost:9876");
producer.start();
Message msg = new Message("sanyouTopic", "TagA", "三友的java日记 ".getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult sendResult = producer.send(msg);

During sending, the producer first pulls the routing table from the NameServer, caches it locally, and refreshes it periodically (default every 30 seconds).

Queue Selection Algorithms

RocketMQ provides two built‑in queue selection strategies:

Round‑robin: messages are distributed evenly across queues (default).

Latency‑fault‑tolerant: the algorithm prefers queues with the smallest recent delivery latency, which may lead to uneven distribution.

Enable the latency‑fault‑tolerant algorithm with:

producer.setSendLatencyFaultEnable(true);

Custom selectors can be implemented by providing a MessageQueueSelector implementation.

SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
    @Override
    public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
        // custom selection logic
        return null;
    }
}, new Object());

Special Cases

If a broker crashes or a send times out, RocketMQ automatically retries (default two times). The retry count can be increased:

producer.setRetryTimesWhenSendFailed(10);

When a message exceeds the default size limit (4 KB), RocketMQ compresses it before sending to reduce network usage.

Message Storage

After a broker receives a message, it persists the data to a file called the CommitLog. Each broker has a single CommitLog that is split into 1 GB segments.

High‑Performance Read/Write

RocketMQ uses zero‑copy techniques (mmap and sendfile) to minimize CPU copying and context switches.

mmap

Memory‑maps a file directly into the process address space, allowing the kernel buffer and application buffer to share data.

FileChannel fileChannel = new RandomAccessFile("test.txt", "rw").getChannel();
MappedByteBuffer mappedByteBuffer = fileChannel.map(FileChannel.MapMode.READ_WRITE, 0, fileChannel.size());

sendfile

Transfers file data directly from kernel space to the socket buffer, reducing both copies and context switches.

FileChannel channel = FileChannel.open(Paths.get("./test.txt"), StandardOpenOption.WRITE, StandardOpenOption.CREATE);
channel.transferTo(position, len, target);

RocketMQ internally relies on the mmap‑based zero‑copy implementation for its own file I/O.

CommitLog Mechanics

Messages are appended sequentially to the CommitLog. When a segment fills up, a new file is created.

Flushing to Disk

Two flushing modes are available:

Asynchronous flushing : the broker writes to the page cache and returns success immediately; a background thread flushes to disk at fixed intervals (default 0.5 s) or after each write.

Synchronous flushing : the broker waits for the data to be flushed to disk before acknowledging the producer, providing higher durability at the cost of throughput.

High Availability

RocketMQ supports two HA modes for broker groups:

Master‑slave synchronous mode : the master (BrokerId 0) synchronously replicates messages and topic metadata to slaves.

Dledger mode : based on the Raft protocol, a leader is elected; the leader writes messages and replicates them to followers. A message is considered committed once a majority of nodes have stored it.

Message Consumption

Consumers retrieve routing information from the NameServer, learn which queues belong to the subscribed topic, and then pull messages from those queues.

Consumption Models

Clustering mode : each queue is assigned to a single consumer within a consumer group, guaranteeing that each message is processed only once.

Broadcasting mode : every consumer in the group receives all queues, so each message is processed by every consumer.

consumer.setMessageModel(MessageModel.BROADCASTING);

ConsumeQueue

To avoid scanning the entire CommitLog, each queue has an associated ConsumeQueue that stores the offset and length of each message in the CommitLog (20 bytes per entry). When a consumer requests messages, the broker looks up the ConsumeQueue to find the exact location in the CommitLog.

Ordered Consumption

Message order is guaranteed when:

The producer always selects the same queue for ordered messages.

The broker stores messages in the ConsumeQueue in the order they arrive.

The consumer processes messages sequentially, typically by using MessageListenerOrderly.

consumer.registerMessageListener(new MessageListenerOrderly() {
    @Override
    public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
        // ordered processing logic
        return null;
    }
});

Message Cleanup

CommitLog files are deleted based on several conditions:

Manual deletion.

Automatic daily cleanup at 4 AM.

When disk usage exceeds 75 % (expire files older than 72 hours are removed).

When disk usage exceeds 85 % (all files are removed regardless of age).

Even unconsumed messages are removed because cleanup operates on the underlying CommitLog files.

Lifecycle Summary

Message Sending : producer creates a message, fetches routing info, selects a queue, and sends the message to the broker.

Message Storage : broker writes the message to CommitLog and records its position in ConsumeQueue; zero‑copy (mmap) is used for high‑performance I/O.

High Availability : in cluster mode, the message is replicated to slave brokers; Dledger provides automatic leader election.

Message Consumption : consumer pulls routing info, selects queues according to its consumption model, reads offsets from ConsumeQueue, then fetches the actual message from CommitLog.

Message Cleanup : CommitLog files are periodically removed based on age and disk‑space thresholds.

https://github.com/sanyou3/rocketmq.git