How RocketMQ Tackles Message Ordering, Duplication, and Transaction Challenges

Background

Distributed messaging systems are essential for building scalable, elastic applications, and they must provide high throughput and strong availability. Two fundamental challenges that any message system must address are message ordering and message duplication.

Key Features and Implementation Principles

1. Ordered Messages

Ordered delivery means consumers receive messages in the same sequence they were produced, e.g., an order’s creation, payment, and completion events. A naïve design that sends each message to a different server cannot guarantee order because network latency may cause later messages to arrive first. RocketMQ solves this by routing all messages of the same logical order (identified by an OrderId) to the same queue and ensuring the producer sends the next message only after the previous one is acknowledged. This guarantees that the first message reaches the broker before the second, preserving order. However, this strict one‑to‑one‑to‑one model reduces parallelism and can become a throughput bottleneck, and it introduces failure‑handling complexities when a consumer does not acknowledge a message.

2. Message Duplication

Duplication arises from network unreliability; a message may be retransmitted or a consumer may process the same message twice after a failure. RocketMQ recommends two mitigation strategies: (1) make consumer business logic idempotent so repeated processing yields the same result, and (2) assign a globally unique identifier to each message and maintain a de‑duplication log that records successfully processed IDs. The second strategy can be implemented either by the broker or by the application, but it may impact throughput, so most systems handle de‑duplication at the business layer.

3. Transaction Messages

Transactional messaging ensures that a local business operation (e.g., a bank transfer) and the corresponding message are either both committed or both rolled back. RocketMQ implements a three‑phase protocol: the producer first sends a Prepared message, then executes the local transaction, and finally sends a Commit or Rollback based on the transaction outcome. The broker periodically scans for Prepared messages and invokes the producer’s checkTransactionState callback to decide the final state. This design guarantees atomicity between the message and the local transaction, but the feature was removed in version 3.2.6, as noted in the GitHub issues linked at the end of the article.

4. Producer Message Sending

When a producer publishes a message, RocketMQ first obtains routing information for the target topic. It then selects a queue using a round‑robin algorithm to achieve load balancing across all queues of that topic. The producer lifecycle includes a start call that (1) discovers name‑server addresses, (2) launches periodic tasks for name‑server updates, broker heartbeats, and topic route refreshes, and (3) starts the load‑balancing service. If sending fails, the producer retries while the number of attempts is less than retryTimesWhenSendFailed and the total elapsed time stays below sendMsgTimeout. After each retry, it may switch to a different queue.

5. Message Storage

RocketMQ stores messages in two complementary files: the ConsumeQueue (a logical index) and the CommitLog (the physical log). The ConsumeQueue contains fixed‑length entries (offset, size, tag hash) that point to the actual message bytes in the CommitLog. Each topic‑queue pair has its own ConsumeQueue file, and special queues such as %RETRY% and %DLQ% are created per consumer group for retry and dead‑letter handling. The CommitLog is a sequential write file shared by all queues on a broker; its entries have variable length and include fields such as offset, message size, and tag hash for efficient filtering.

6. Message Subscription

RocketMQ supports both push and pull subscription modes, but both are implemented as the consumer actively pulling messages from the broker. A RebalanceService thread runs every ten seconds to evenly distribute topic queues among consumers in the same consumer group, using strategies like average allocation. In push mode, the broker holds the consumer’s pull request and returns data as soon as messages become available (long‑polling). If no data is present, the request is blocked until a timeout or new data arrives.

7. Other Features

Timed (delayed) messages

Configurable flush policies (synchronous double‑write, asynchronous replication)

Massive message accumulation capability

High‑efficiency communication components delivering millisecond‑level latency

Best Practices

Producer

Use a single topic per application and differentiate sub‑types with tags.

Set a unique business key in the keys field for traceability.

Log send results and keys for debugging.

Implement retry mechanisms (store failed messages in a DB and retry later).

For fire‑and‑forget scenarios, use sendOneWay.

Consumer

Ensure idempotent processing to avoid duplicate effects.

Prefer batch consumption to improve throughput.

Optimize per‑message handling logic.

Configuration

Disable autoCreateTopicEnable in production to prevent the broker from automatically creating topics on the first write, which can lead to uneven load distribution.

Design Assumptions

Any node may fail; the system must tolerate single‑machine outages.

Cluster capacity may be insufficient under load.

Worst‑case scenarios are expected to occur.

Low latency is required for a good user experience in internal networks.

Key design goals of RocketMQ include distributed clustering, strong data safety, massive data storage, and millisecond‑level delivery.

References

https://github.com/alibaba/RocketMQ/issues/65

https://github.com/alibaba/RocketMQ/issues/138

https://github.com/alibaba/RocketMQ/issues/156