Message Deduplication and Exactly‑Once Semantics in RocketMQ

Message middleware is a core component of distributed systems, providing asynchronous communication, decoupling, and traffic shaping, and it guarantees that a message will not be lost once successfully delivered to the broker.

The most basic reliability guarantee is the at‑least‑once semantics, meaning each message is guaranteed to be consumed by a consumer at least once, which can lead to duplicate deliveries.

RocketMQ may deliver duplicate messages in three situations: (1) duplicate sending due to network glitches or client crashes after the broker has persisted the message, (2) duplicate delivery when the consumer’s acknowledgment fails, and (3) duplicate delivery caused by load‑balancing events such as broker or consumer restarts.

A simple deduplication method is to wrap the business logic (e.g., inserting an order and updating inventory) in a database transaction and check for existing records before processing, as shown in the following SQL snippet:

insert into t_order values ...; update t_inv set count = count-1 where good_id = 'good123';

To make the processing idempotent, the code can first query the order table and return early if the record already exists:

select * from t_order where order_no = 'order123'; if (order != null) { return; // duplicate message }

However, this approach suffers from concurrency issues; two threads may both see no record and proceed, causing duplicate processing. Using SELECT ... FOR UPDATE within a transaction can serialize access but reduces concurrency.

The article then introduces the concept of Exactly‑Once semantics, where a message is processed exactly once even if the producer retries, which is difficult to achieve in distributed environments without coordinated state between the broker, client, and consumer.

For transaction‑based deduplication, the workflow is: start a transaction, insert a deduplication record (with a unique key), update the business table, and commit. This ensures that even if the broker retries, the insert will fail due to primary‑key conflict, preventing duplicate processing.

Limitations of the transaction‑based method include reliance on relational‑database transactions, inability to roll back non‑transactional resources (e.g., Redis), and challenges with cross‑database scenarios.

To overcome these limits, a non‑transactional approach using a dedicated deduplication table (or Redis) with a status field (processing / completed) is proposed. The table records the message ID and a TTL (e.g., 10 minutes) to expire stale processing entries.

When a message arrives, the consumer attempts to insert the deduplication record; if the insert succeeds, it proceeds with business logic; if it fails due to a primary‑key conflict, the message is considered a duplicate and either delayed or discarded.

Using Redis simplifies the implementation because its native TTL handles expiration, and the overhead is lower than MySQL, though consistency guarantees are weaker.

A Java example of integrating the Redis‑based deduplication listener with RocketMQ is provided:

DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("TEST-APP1"); consumer.subscribe("TEST-TOPIC", "*"); String appName = consumer.getConsumerGroup(); StringRedisTemplate stringRedisTemplate = null; // obtain from Spring context DedupConfig dedupConfig = DedupConfig.enableDedupConsumeConfig(appName, stringRedisTemplate); DedupConcurrentListener messageListener = new SampleListener(dedupConfig); consumer.registerMessageListener(messageListener); consumer.start();

The article concludes that while the Redis‑based deduplication solves the majority of duplicate‑message problems (broker‑induced and producer‑induced), it does not fully guarantee idempotency in all failure scenarios, especially when intermediate RPC calls are not idempotent.

Best‑practice recommendations include: ensuring business operations are idempotent or have rollback mechanisms, handling graceful consumer shutdowns, and monitoring message retries to intervene manually when necessary.