Why Does RocketMQ Duplicate Message Consumption? Root Causes Explained

Message Send Exceptions Leading to Retries

When a producer encounters a send timeout or no response, RocketMQ retries the send operation (default up to two times) by selecting another broker’s queue. If the original broker actually processed the message but the producer did not receive a response, the retry causes duplicate messages, which the consumer will later consume repeatedly.

Consumer Exceptions Triggering Re‑consumption

In concurrent consumption, users implement the MessageListenerConcurrently interface. If processing a batch of messages throws an exception, the listener returns RECONSUME_LATER, causing the entire batch to be re‑delivered. Because the batch may contain successfully processed messages, those messages are consumed again, leading to duplication. The default batch size is 1, but increasing it raises the risk.

Consumer Offset Commit Failures

After processing, a consumer must commit the offset to RocketMQ. Offsets are stored in memory and periodically (default every 5 seconds) sent to the broker. If the broker crashes before receiving the latest offset, the consumer will restart from an older offset, causing the messages between the lost offset and the actual processing point to be consumed again.

Broker Offset Persistence Failures

The broker also persists offsets to disk every 5 seconds. A broker restart after a crash may lose the most recent offsets, so consumers receive a lower offset from the broker and re‑consume messages that were already processed.

Master‑Slave Offset Synchronization Failures

In high‑availability mode, the slave synchronizes metadata (including offsets) from the master every 10 seconds. If the master fails, the slave may miss up to 10 seconds of offset updates, leading to consumers receiving stale offsets from the new master and re‑processing messages.

Rebalance Causing Duplicate Consumption

When a consumer group’s membership changes, RocketMQ rebalances queue assignments. If a consumer is removed from a queue during rebalance, its uncommitted offsets are lost, so the newly assigned consumer starts from an older offset, causing the same messages to be consumed again.

Long‑Running Message Cleanup

RocketMQ runs a task every 15 minutes that removes messages that have been in the processing set for too long. If a message is removed before its offset is committed, the consumer may later re‑receive that message, resulting in duplicate consumption.

Conclusion

RocketMQ can duplicate message consumption due to send retries, consumer‑side exceptions, offset commit or persistence issues, master‑slave synchronization gaps, rebalance events, and long‑processing cleanup. To guarantee exactly‑once processing, applications must implement appropriate idempotency mechanisms tailored to their business scenarios.