How to Guarantee Exactly‑Once Message Consumption in High‑Concurrency Systems

Introduction

During a recent interview at Alibaba, a candidate was asked how to ensure a message is consumed only once in a high‑concurrency scenario, requiring a full‑stack analysis and concrete solutions.

Business Scenarios

When using a message queue, duplicate consumption can cause issues such as multiple coupon deliveries, duplicate payment notifications, or repeated shipments.

Root Causes of Duplicate Consumption

Producer retries due to network jitter or missing ACK.

Broker failover causing undelivered messages to be replayed.

Consumer crashes after processing but before committing offset.

Long processing time leading to rebalance and re‑assignment.

Three‑Layer Defense Strategy

A message passes through producer, broker, and consumer stages; each layer can implement safeguards to prevent duplication.

3.1 Producer‑Side Protection

Idempotent sending (e.g., Kafka or Pulsar with enable.idempotence=true).

Transactional messages.

<code>Properties props = new Properties();
props.put("enable.idempotence", "true"); // enable idempotence
props.put("acks", "all");               // all replicas must acknowledge
KafkaProducer&lt;&gt; producer = new KafkaProducer<>(props);
</code>

<code>Send Half Message (invisible to consumers).
Execute local transaction (e.g., update order status).
Commit or rollback the message based on transaction result.
</code>

3.2 Broker‑Side De‑duplication and Stable Delivery

Attach a unique message key (e.g., order ID) for broker‑side deduplication.

Ensure persistence and ordering (Kafka acks=all, RocketMQ SYNC_FLUSH, partitioning by business ID, Raft/ISR replication).

3.3 Consumer‑Side Idempotency

Database unique constraints.

Optimistic locking.

State‑machine validation.

Redis distributed lock.

Deduplication table.

<code>-- Insert order, fails if order_id already exists
INSERT INTO orders (order_id, user_id, amount, status)
VALUES ('20231001123456', 1001, 99.99, 'UNPAID');
</code>

<code>-- Optimistic lock example
UPDATE account SET balance = balance - 100, version = version + 1
WHERE user_id = 123 AND version = 5;
</code>

<code>-- State‑machine check
UPDATE orders SET status = 'PAID'
WHERE order_id = '20231001123456' AND status = 'UNPAID';
</code>

<code>// Redisson distributed lock example
RLock lock = redisson.getLock("MSG_LOCK:" + messageId);
if (lock.tryLock(10, 30, TimeUnit.SECONDS)) {
    try {
        if (isProcessed(messageId)) return;
        process(message);
        markAsProcessed(messageId);
    } finally {
        lock.unlock();
    }
}
</code>

Fallback: Monitoring + Reconciliation

Because 100 % guarantee is unrealistic, add metrics such as producer resend rate, consumer duplicate alerts, and offset commit latency, plus periodic reconciliation between message counts and business data.

Why Not Use Exactly‑Once?

Exactly‑once (e.g., RocketMQ’s consumption mode) incurs higher latency and complexity; most scenarios prefer At‑Least‑Once combined with idempotent processing.

Conclusion

In everyday development, ensuring no duplicate consumption mainly relies on robust idempotent design on the consumer side, while the three‑layer defense and monitoring provide additional safety, especially for interview discussions.