How to Achieve Exactly‑Once Message Processing in RocketMQ Without Heavy Transactions

Message middleware is a core component of distributed systems, providing asynchronous processing, decoupling, and traffic shaping. It is typically considered reliable: once a message is successfully delivered to the broker, it will not be lost and will be consumed at least once (the "AT LEAST ONCE" guarantee).

Because of this reliability, the same message may be delivered multiple times. If a consumer crashes after processing a message but before acknowledging it, the broker will redeliver the message, leading to duplicate consumption.

Simple Message Deduplication Solution

Assume the consumer inserts an order record and updates inventory:

insert into t_order values ...
update t_inv set count = count-1 where good_id = 'good123';

To make the operation idempotent, one might first check whether the order already exists:

select * from t_order where order_no = 'order123';
if (order != null) {
    return; // duplicate, skip
}

This works in many cases but fails under high concurrency because the check and the insert are not atomic.

Concurrent Duplicate Messages

When duplicate messages arrive within the processing window, the check may still see an empty result, allowing the duplicate to proceed and cause issues such as primary‑key conflicts or double inventory deduction.

One Concurrency‑Safe Solution

Wrap the check in a transaction using SELECT ... FOR UPDATE to lock the row:

select * from t_order where order_no = 'THIS_ORDER_NO' for update;
if (order.status != null) {
    return; // duplicate, skip
}

While safe, this approach reduces throughput because the transaction holds locks.

Exactly‑Once

Exactly‑Once means a message is processed successfully exactly one time, even if the producer retries. In practice, achieving this universally is impossible, but it is feasible when the consumer logic relies on a relational‑database transaction.

Transactional Insertion into a Message Table

Insert a record into a dedicated message‑consumption table within the same transaction that updates the business data:

Begin transaction

Insert into message table (handle primary‑key conflict)

Update order table (original business logic)

Commit transaction

If the transaction commits, the message table entry guarantees the message is considered consumed; if the transaction fails, the broker will redeliver the message.

Limitations:

The consumption logic must be confined to a relational‑database transaction.

All involved data must reside in the same database; cross‑database scenarios are not covered.

Non‑Transactional Deduplication Using a Message Idempotence Table

Instead of relying on a transaction, store a consumption status (e.g., "processing", "completed") in a message table. Only messages marked "completed" are considered idempotent; others trigger delayed retries.

When using MySQL, the unique primary key ensures only one insert succeeds, and subsequent duplicates fail, causing the broker to retry later.

To avoid endless retries when a message remains in "processing" due to a failure, set an expiration (e.g., 10 minutes) and clean up stale entries.

Using Redis as the Idempotence Store

Redis offers lower latency and built‑in TTL for expiration, making it a suitable alternative to MySQL for the idempotence table, though it sacrifices some durability guarantees.

Sample Code (RocketMQ Dedup Listener)

// Using Redis for idempotence
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("TEST-APP1");
consumer.subscribe("TEST-TOPIC", "*");
String appName = consumer.getConsumerGroup();
StringRedisTemplate stringRedisTemplate = null; // obtain template elsewhere
DedupConfig dedupConfig = DedupConfig.enableDedupConsumeConfig(appName, stringRedisTemplate);
DedupConcurrentListener messageListener = new SampleListener(dedupConfig);
consumer.registerMessageListener(messageListener);
consumer.start();

The only modification to standard RocketMQ code is the creation of a DedupConcurrentListener that handles idempotence based on a business key (defaulting to the messageId).

Is This a One‑Size‑Fits‑All Solution?

While the approach solves the majority of duplicate‑message problems, it does not cover scenarios where a consumer crashes after partially completing non‑idempotent operations (e.g., RPC calls, external locks). In such cases, additional compensation or rollback mechanisms are required.

Practical Value

The method effectively handles:

Broker‑induced duplicate deliveries.

Producer‑induced duplicate messages.

Concurrent duplicate consumption windows.

These cover roughly 99% of real‑world duplicate‑message cases.

Additional Recommendations

Implement rollback for failed consumption when possible.

Ensure graceful shutdown of consumers to avoid mid‑process crashes.

Make non‑idempotent operations abort and alert on failure.

Monitor consumption retries and manually intervene when necessary.