Message Deduplication and Exactly-Once Semantics in RocketMQ: Strategies and Implementation

Message middleware such as RocketMQ provides at‑least‑once delivery, which guarantees that a message will be consumed at least once but may be delivered multiple times, leading to duplicate processing problems.

A simple deduplication approach checks the business table (e.g., order table) before processing, but this can fail under high concurrency because the check may see an empty record while another consumer is still processing the same message.

Concurrent solutions include using SELECT FOR UPDATE to lock the row or applying optimistic locking, both of which increase transaction time and reduce concurrency.

The article introduces the concept of Exactly Once semantics—messages are processed successfully exactly one time—and explains why achieving this generally requires reliance on relational‑database transactions.

update t_order set status = 'SUCCESS' where order_no='order123';

By inserting a dedicated message‑consumption record table and wrapping the insert and business updates in a single transaction, the system can guarantee that a message is processed only once while still preserving at‑least‑once delivery.

Limitations of this transaction‑based method include the need for all operations to be within the same relational database and the inability to roll back non‑transactional resources such as Redis.

A non‑transactional alternative stores a consumption status (e.g., processing , completed ) in a message table. Only messages marked completed are considered deduplicated; others trigger delayed retries, preventing loss while handling duplicates.

select * from t_order where order_no='order123'

Using Redis as the storage medium further reduces latency and leverages TTL for automatic expiration of stale processing states, though it sacrifices the strong consistency guarantees of MySQL.

The article provides a ready‑to‑use Java implementation for RocketMQ deduplication. The core change is the registration of a DedupConcurrentListener that handles idempotent consumption based on a configurable business key (defaulting to messageId ).

// Sample usage of DedupConcurrentListener with Redis
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("TEST-APP1");
consumer.subscribe("TEST-TOPIC", "*");
String appName = consumer.getConsumerGroup();
StringRedisTemplate stringRedisTemplate = null; // obtain Redis template
DedupConfig dedupConfig = DedupConfig.enableDedupConsumeConfig(appName, stringRedisTemplate);
DedupConcurrentListener messageListener = new SampleListener(dedupConfig);
consumer.registerMessageListener(messageListener);
consumer.start();

While this solution does not eliminate all edge‑case failures (e.g., crashes during a partially processed transaction), it effectively solves the majority of duplicate‑message scenarios caused by broker retries, producer re‑sends, and concurrent consumption.

Additional recommendations include implementing rollback mechanisms for failed consumptions, ensuring graceful consumer shutdowns, making non‑idempotent operations fail fast with alerts, and monitoring retry patterns to intervene manually when necessary.