How to Achieve Exactly-Once Message Processing with RocketMQ Deduplication

Message middleware is a common component in distributed systems, widely used for asynchronous processing, decoupling, and peak shaving.

It is usually considered reliable: once a message is successfully delivered to the middleware, it will not be lost.

At least once is the basic guarantee: the message will be consumed successfully at least once.

This "AT LEAST ONCE" semantics means the message will be "successfully consumed at least once".

When a consumer crashes before acknowledging a message, the middleware will keep redelivering the same message until it is acknowledged, which can lead to duplicate deliveries.

Three Duplicate Scenarios

RocketMQ documentation lists three cases where duplicate messages may appear:

1. Duplicate on send : After a message is persisted, a network glitch or client crash causes the server to fail to respond. If the producer retries, consumers receive two identical messages with the same Message ID.

2. Duplicate on delivery : After a consumer processes a message, a network glitch occurs when sending the acknowledgment. The server retries delivery after recovery, causing duplicate consumption.

3. Duplicate during load balancing (e.g., broker or consumer restart): Rebalancing can cause consumers to receive duplicate messages.

Simple Deduplication Solutions

Assume the consumption logic inserts an order record and updates inventory:

insert into t_order values ...;
update t_inv set count = count-1 where good_id = 'good123';

A basic idempotent approach checks whether the order already exists:

select * from t_order where order_no = 'order123';
if (order != null) {
    return; // duplicate, skip
}

This works in many cases but fails under high concurrency.

Concurrent Duplicate Messages

When two threads process the same message within a short interval, both may see the order as absent and proceed, causing duplicate inserts or inventory deductions.

Using a transaction with SELECT ... FOR UPDATE locks the row and prevents the race:

select * from t_order where order_no = 'THIS_ORDER_NO' for update;
if (order.status != null) {
    return; // duplicate
}

However, wrapping the whole consumption in a transaction reduces concurrency.

Exactly Once

Exactly Once means a message is processed and persisted exactly one time, even if the producer retries.

Exactly-Once is that a message sent to the system can be processed by the consumer only once, even if the producer retries.

Achieving this generally requires coordination among the broker, client, and consumer logic, which is difficult in distributed environments.

Transaction‑Based Exactly Once (Database)

Insert a consumption record into a dedicated message table within the same transaction that updates the business data:

BEGIN;
INSERT INTO message_log ...; -- handle primary‑key conflict
UPDATE t_order SET status='SUCCESS' WHERE order_no='order123';
COMMIT;

If the transaction commits, the message log entry guarantees the message was processed; subsequent redeliveries will fail on primary‑key conflict and be ignored.

This approach works only when the entire consumption can be wrapped in a relational‑database transaction.

Limitations of Transaction‑Based Approach

It cannot handle non‑transactional resources (e.g., Redis), cross‑database operations, or long‑running RPC calls, and it may cause lock contention.

Non‑Transactional Deduplication Using a Message Table

Store messages in a deduplication table with a status field (e.g., IN_PROGRESS, DONE) without relying on a transaction.

When a message is first processed, insert a row; if the insert succeeds, proceed. If a duplicate insert occurs, treat it as a repeat and either delay or discard.

Set an expiration (e.g., 10 minutes) on IN_PROGRESS rows; if processing fails or the service crashes, the row expires and the message can be retried.

Redis‑Based Implementation

Redis can be used as the deduplication store, leveraging its TTL feature for expiration and offering lower latency.

Example code (Java) shows how to enable a deduplication listener with a Redis template:

// Using Redis for deduplication
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("TEST-APP1");
consumer.subscribe("TEST-TOPIC", "*");
String appName = consumer.getConsumerGroup();
StringRedisTemplate stringRedisTemplate = null; // obtain template
DedupConfig dedupConfig = DedupConfig.enableDedupConsumeConfig(appName, stringRedisTemplate);
DedupConcurrentListener messageListener = new SampleListener(dedupConfig);
consumer.registerMessageListener(messageListener);
consumer.start();

Practical Value

The solution effectively handles most duplicate‑delivery cases caused by broker issues, producer retries, and load‑balancing, reducing the need for custom idempotent code in each consumer.

However, it does not fully solve scenarios where a consumer crashes mid‑process, leaving external resources (e.g., locked inventory) in an inconsistent state.

Additional Recommendations

Ensure consumption failures are rolled back whenever possible.

Implement graceful shutdown for consumers to avoid mid‑process crashes.

Make non‑idempotent operations either idempotent or trigger alerts when they cannot be safely retried.

With proper monitoring and fallback mechanisms, the described deduplication approach can resolve the majority of duplicate‑message problems in RocketMQ‑based systems.