How to Achieve Exactly‑Once Message Processing with RocketMQ Deduplication

Message middleware is a common component in distributed systems, providing asynchronous processing, decoupling, and traffic shaping. It is typically considered reliable: once a message is successfully delivered to the middleware, it will not be lost and will be consumed at least once (the “AT LEAST ONCE” guarantee).

For example, if message M is delivered to consumer A, A receives it and starts processing, but the program restarts before acknowledging success, the middleware will redeliver the message until consumption succeeds.

This reliability can cause duplicate deliveries. If A finishes processing M, attempts to acknowledge success, and then restarts, the middleware sees the message as unacknowledged and will deliver it again, leading to apparent duplicate consumption.

Simple Message Deduplication Solution

Assume the consumption logic inserts an order record and updates inventory:

insert into t_order values ...
update t_inv set count = count-1 where good_id = 'good123';

A basic idempotent approach checks whether the order already exists:

select * from t_order where order_no = 'order123';
if (order != null) {
    return; // duplicate, skip
}

This works in many cases but fails under high concurrency.

Concurrent Duplicate Messages

If duplicate messages arrive within the processing window (e.g., within 100 ms), the above check may still see an empty result because the first transaction has not yet committed, allowing the duplicate to proceed and cause issues such as primary‑key conflicts or double inventory deduction.

One Concurrency‑Safe Solution

Wrap the select in a transaction using SELECT ... FOR UPDATE to lock the row:

select * from t_order where order_no = 'THIS_ORDER_NO' for update;
if (order.status != null) {
    return; // duplicate, skip
}

While effective, this introduces transaction overhead and reduces concurrency.

Exactly Once

Exactly‑Once means a message is processed by the consumer only once, even if the producer retries and the broker re‑delivers the same message.

Achieving a true exactly‑once guarantee is difficult in distributed environments, but it is feasible when the consumption logic relies on relational‑database transactions.

Transactional Insertion into a Message Table

Insert a record into a dedicated message‑consumption table within the same transaction as the business update:

Begin transaction

Insert into the message table (handle primary‑key conflicts)

Update the order table (original business logic)

Commit transaction

If the transaction commits, the message table entry guarantees that subsequent re‑deliveries will fail on primary‑key conflict, preventing duplicate processing.

If the service crashes before committing, neither the order nor the message table is updated, so the broker will retry and the message will be processed correctly.

Limitations: the consumption logic must be fully encapsulated in a relational‑database transaction; non‑transactional resources (e.g., Redis) cannot be rolled back, and cross‑database operations are unsupported.

More Complex Business Scenarios

When a workflow involves multiple RPC calls, inventory locks, and other non‑transactional steps, a single database transaction cannot guarantee atomicity. In such cases, developers often fall back to explicit locking (e.g., SELECT FOR UPDATE) or optimistic‑lock techniques.

Generalized Non‑Transactional Deduplication Using a Message Table

Instead of relying on a transaction, maintain a message table with a consumption status (e.g., "processing", "completed"). Only messages marked as completed are considered deduplicated. In‑flight messages trigger delayed retries, preventing premature acknowledgment.

Diagram of the non‑transactional flow:

To avoid lost messages in the delayed‑retry path, each in‑flight record includes an expiration time (e.g., 10 minutes). If a message remains in the "processing" state beyond this window, it is removed so that a subsequent retry can be treated as a fresh attempt.

Flexible Storage for the Message Table

The deduplication table can be stored in a fast key‑value store such as Redis, which offers lower latency and built‑in TTL for expiration, though it sacrifices the strong consistency guarantees of a relational database.

Source Code: RocketMQDedupListener

The Java implementation for RocketMQ is open‑source on GitHub. A Redis‑based usage example:

// Use Redis as the idempotent table
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("TEST-APP1");
consumer.subscribe("TEST-TOPIC", "*");

String appName = consumer.getConsumerGroup(); // usually the consumer group name
StringRedisTemplate stringRedisTemplate = null; // obtain StringRedisTemplate elsewhere
DedupConfig dedupConfig = DedupConfig.enableDedupConsumeConfig(appName, stringRedisTemplate);
DedupConcurrentListener messageListener = new SampleListener(dedupConfig);

consumer.registerMessageListener(messageListener);
consumer.start();

The only modification to standard RocketMQ code is the creation of a DedupConcurrentListener that specifies the deduplication key (default is messageId).

Is This a One‑Size‑Fits‑All Solution?

While the approach solves the majority of duplicate‑message problems, it does not fully address scenarios where a consumer crashes after partially processing a message (e.g., after locking inventory). In such cases, the subsequent retry will re‑execute non‑idempotent steps unless those steps themselves are made idempotent or guarded by additional checks.

Value of This Implementation

It reliably handles most duplicate‑delivery cases caused by broker retries or producer re‑sends, and it prevents concurrent duplicate consumption from entering business logic simultaneously.

Additional Deduplication Recommendations

Ensure failed consumption rolls back any side effects.

Implement graceful consumer shutdown to avoid mid‑process crashes.

Make non‑idempotent operations either idempotent or abort the transaction with an alarm.

Monitor consumption failures and manually roll back when necessary.