Production-Ready Idempotency for RocketMQ Duplicate Consumption (Full Code)

Why Duplicate Consumption Is Not a Bug

RocketMQ uses an At‑Least‑Once delivery model. Network jitter, broker master‑slave failover, consumer restart or rebalance can cause the same message to be delivered multiple times.

Idempotency vs. No Duplicates

Idempotency means that repeated execution yields the same business result, not that the operation never repeats. The correct design principle is to allow duplicates but guarantee a single correct outcome .

Root Causes of Duplicate Consumption (Production‑Level View)

1️⃣ Producer Side (Uncontrollable)

Synchronous or asynchronous send failures trigger automatic retries.

Transactional message status checks may be uncertain.

2️⃣ Consumer Side (High Frequency)

Process crashes before the ACK is sent.

Consumer restart causes offset rollback.

Rebalance redistributes queues, causing re‑delivery.

3️⃣ Broker Layer (System Level)

Master‑slave switch.

Consumption timeout triggers redelivery.

Conclusion: Duplicate consumption cannot be eliminated; it must be handled in the consumer.

First Defense Line – Redis Idempotency (Fast Deduplication)

Applicable Scenarios

High‑frequency messages.

Latency‑sensitive processing.

Redis data loss is tolerable (Redis is used as a cache, not the source of truth).

1️⃣ Quick Duplicate Check with SETNX (Recommended)

@Component
public class RedisIdempotentService {

    private static final String PREFIX = "msg:idempotent:";
    private static final long EXPIRE_SECONDS = 24 * 3600; // 1 day

    @Autowired
    private StringRedisTemplate redisTemplate;

    /**
     * Returns true only on the first call for a given msgId.
     */
    public boolean checkAndSet(String msgId) {
        Boolean success = redisTemplate.opsForValue()
                .setIfAbsent(PREFIX + msgId, "1", EXPIRE_SECONDS, TimeUnit.SECONDS);
        return Boolean.TRUE.equals(success);
    }

    public void remove(String msgId) {
        redisTemplate.delete(PREFIX + msgId);
    }
}

2️⃣ Production‑Grade Considerations

SETNX success only guarantees that the key was created; it does not guarantee that the business transaction succeeded. A safer pattern uses two keys with different TTLs:

PROCESSING  // short TTL, indicates the message is being processed
SUCCESS     // long TTL, indicates successful processing

If the process is OOM‑killed before writing the SUCCESS key, the business result may be lost. Therefore the final correctness must be enforced downstream (e.g., database).

Second Defense Line – Database Idempotency (Core Guarantee)

Redis is an optimization layer; the database provides true correctness.

1️⃣ Unique Constraints (Most Reliable)

CREATE TABLE message_consume_log (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    message_id VARCHAR(64) NOT NULL,
    business_key VARCHAR(128) NOT NULL,
    create_time DATETIME DEFAULT CURRENT_TIMESTAMP,
    UNIQUE KEY uk_msg_id (message_id),
    UNIQUE KEY uk_business_key (business_key)
);

It is recommended to define a business‑level unique key (e.g., order number) rather than relying solely on the MQ messageId.

2️⃣ Using the Unique Key for Idempotency

try {
    consumeLogDao.insert(messageId, orderNo); // unique on business_key
    // continue business logic
} catch (DuplicateKeyException e) {
    // Record already exists → treat as already processed
    log.info("Duplicate message ignored: {}", messageId);
    // ACK the message
}

Third Defense Line – State Machine + Conditional Update (Ultimate Solution)

All core business flows should be modelled as a state machine.

1️⃣ Irreversible State Design

WAIT_PAY → PAID → FINISHED

Transitions that move backward (e.g., PAID → WAIT_PAY) must be prohibited.

2️⃣ Conditional Update SQL (Optimistic Locking)

UPDATE order_info
SET status = 'PAID',
    version = version + 1
WHERE order_no = #{orderNo}
  AND status = 'WAIT_PAY'
  AND version = #{version};

The version column prevents lost updates when multiple consumers try to process the same order.

3️⃣ Java Transactional Example

@Transactional
public void processOrder(OrderMessage msg) {
    Order order = orderDao.selectByOrderNo(msg.getOrderNo());

    if (order.getStatus() != OrderStatus.WAIT_PAY) {
        // Already processed – idempotent return
        return;
    }

    int rows = orderDao.updateStatus(
            msg.getOrderNo(),
            OrderStatus.WAIT_PAY,
            OrderStatus.PAID,
            order.getVersion()
    );

    if (rows == 0) {
        // State changed concurrently → treat as duplicate
        throw new IdempotentException("State has changed");
    }

    // Subsequent business logic (e.g., inventory deduction)
}

Complete Layered Protection Architecture

RocketMQ
   ↓
Redis Idempotency (fast interception)
   ↓
Business State Machine (logic validation)
   ↓
Database Unique Constraints (final safeguard)
   ↓
Business Processing

Full Consumer Example (Production‑Ready)

@RocketMQMessageListener(
        topic = "order_topic",
        consumerGroup = "order_group"
)
@Component
public class OrderConsumer implements RocketMQListener<MessageExt> {

    @Autowired
    private RedisIdempotentService redisService;
    @Autowired
    private OrderService orderService;

    @Override
    public void onMessage(MessageExt msg) {
        String msgId = msg.getMsgId();
        OrderMessage orderMsg = parse(msg);

        // First layer: Redis idempotency
        if (!redisService.checkAndSet(msgId)) {
            // Duplicate detected → ignore
            return;
        }

        try {
            orderService.processOrder(orderMsg);
        } catch (Exception e) {
            // Remove the Redis key so the message can be retried
            redisService.remove(msgId);
            throw e; // trigger RocketMQ retry
        }
    }
}

ACK and Retry Best Practices (Advanced)

Already processed (idempotent hit) → ACK.

Unique‑key conflict → ACK.

Business exception (e.g., validation failure) → do NOT ACK, let RocketMQ retry.

System exception (e.g., DB outage) → do NOT ACK, let RocketMQ retry.

Irrecoverable exception → send to DLQ.

Known duplicate = successful consumption.

Monitoring and Alerts (Beyond Code)

Duplicate consumption rate (messages flagged by Redis).

Redis key count (size of the idempotency cache).

Database unique‑key conflict count.

Message backlog depth.

DLQ message volume.

Final Takeaway

RocketMQ’s duplicate consumption is inherent to its At‑Least‑Once semantics. The real problem is the lack of idempotent design. Production systems should: Accept that duplicates will occur. Design idempotent processing (Redis fast check → state machine → DB unique constraint). Ensure the final business state moves only forward. Messages can be replayed, but business state must never regress.