How to Achieve Exactly-Once Message Processing with RocketMQ Deduplication

Message middleware is a common component in distributed systems, providing features such as asynchronous processing, decoupling, and peak‑shaving, but it guarantees only at‑least‑once delivery.

At‑least‑once means a message will be delivered to a consumer at least once, which can lead to duplicate deliveries.

When a consumer crashes after processing a message but before acknowledging it, the middleware will redeliver the same message, causing apparent duplicate consumption. In RocketMQ this appears as repeated messages with the same messageId.

Simple message deduplication solution

Assume the business logic inserts an order record and updates inventory. A naïve idempotent check might query the order table and return early if the record already exists:

select * from t_order where order_no = 'order123'
if (order != null) {
    return; // duplicate, skip
}

This works in many cases but fails under high concurrency because two threads may both see no record and proceed.

Concurrent duplicate messages

To handle concurrency, the select can be changed to SELECT ... FOR UPDATE inside a transaction, locking the row:

select * from t_order where order_no = 'THIS_ORDER_NO' for update
if (order.status != null) {
    return; // duplicate, skip
}

While effective, wrapping the entire consumer logic in a transaction reduces throughput.

Exactly‑Once

The ideal delivery semantics is Exactly‑Once : a message is processed exactly one time even if the producer retries. In practice, achieving this requires coordination between the broker, client library, and consumer logic, and is rarely fully realized.

Based on relational‑database transaction inserting a message table

One approach is to create a dedicated message‑consumption table and perform the original business update and the insertion of a consumption record in the same database transaction:

1. Begin transaction 2. Insert consumption record (handle primary‑key conflict) 3. Update business table 4. Commit transaction

If the transaction commits, the consumption record guarantees that the message will not be processed again, even if the broker later retries. This method relies entirely on relational‑database ACID guarantees.

More complex business scenarios

When the processing flow involves multiple RPC calls, external services, or cross‑database operations, a single database transaction cannot cover the whole workflow, and non‑transactional resources cannot be rolled back.

More generic solution

Instead of relying on a transaction, we can use a message‑idempotent table that stores a consumption status (e.g., processing , completed ). Insertion succeeds for the first attempt; subsequent attempts fail due to a primary‑key conflict and are redirected to a delayed‑retry path. The status field prevents concurrent processing of the same message.

Using Redis as the storage medium provides low latency and built‑in TTL for expiring stale “processing” entries.

Show me code

// Use Redis for idempotent table
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("TEST-APP1");
consumer.subscribe("TEST-TOPIC", "*");

String appName = consumer.getConsumerGroup(); // usually the consumer group name
StringRedisTemplate stringRedisTemplate = null; // obtain a StringRedisTemplate instance
DedupConfig dedupConfig = DedupConfig.enableDedupConsumeConfig(appName, stringRedisTemplate);
DedupConcurrentListener messageListener = new SampleListener(dedupConfig);

consumer.registerMessageListener(messageListener);
consumer.start();

Is this a one‑stop solution?

Even with idempotent tables, failures that occur after the consumption status is set to processing but before the business logic completes can cause the message to be retried and eventually dead‑lettered if the processing never finishes.

Value of this implementation

The approach solves the majority of duplicate‑delivery problems caused by broker retries, producer resends, and load‑balancing events, handling about 99 % of real‑world cases.

Other deduplication recommendations

Ensure failed consumption rolls back any side effects.

Implement graceful consumer shutdown to avoid mid‑process crashes.

Make non‑idempotent operations either abort with an error or be protected by additional idempotent checks.

Monitor consumption failures and manually intervene when necessary.