How to Prevent Message Loss and Duplication in Message Queues

Message Loss

Message loss can occur at three stages: producer send, broker storage, and consumer consumption.

Producer‑side loss

Most message queues support both synchronous and asynchronous sends.

Synchronous send : The producer blocks until it receives an ACK from the broker. If no ACK is received within a timeout, the client must retry. This guarantees delivery but adds latency.

Asynchronous send : The producer supplies a callback that is invoked when the broker replies. The callback must examine both the metadata and any exception to decide whether to retry.

producer.send(record, new Callback() {
    @Override
    public void onCompletion(RecordMetadata metadata, Exception exception) {
        if (exception != null) {
            logger.error("Failed to send message", exception);
            // retry logic here
        } else {
            logger.info("Message sent successfully, offset={}", metadata.offset());
        }
    }
});

Broker storage loss

Even after the producer receives an ACK, a broker crash before persisting the record can cause loss. Two common safeguards are:

Synchronous flush to disk : Configure the broker to wait for the write to be flushed before acknowledging. Example for RocketMQ: flushDiskType=SYNC_FLUSH.

Replication / clustering : Deploy multiple broker nodes and require acknowledgments from a majority (quorum‑based replication). If one node fails, the remaining nodes still retain the data.

Consumer‑side loss

Consumers should acknowledge a message only after the processing is durably persisted (e.g., written to a database). A typical pattern is:

Persist the raw message or its identifier locally.

Execute business logic.

Send the ACK to the broker.

Message Duplication

Duplication arises when a producer does not receive an ACK and retries, or when a broker does not receive the consumer’s ACK and resends the message.

Idempotent consumption

No mainstream queue automatically eliminates duplicates; applications must enforce idempotency.

Database unique constraint : Store the message ID (or another unique business key) in a table with a UNIQUE index. Insertion will fail for a duplicate, allowing the consumer to skip processing.

Redis deduplication : Use SETNX (or setIfAbsent) to record the message ID before processing. If the key already exists, the message is a duplicate.

ValueOperations<String, String> ops = redisTemplate.opsForValue();
Boolean first = ops.setIfAbsent(messageId, "1");
if (Boolean.TRUE.equals(first)) {
    // process the message
} else {
    logger.warn("Duplicate message detected, skipping ID={}", messageId);
}

Note: If processing fails after the ID has been stored, the key should be removed (or set with a short TTL) so that a retry can be performed.

Summary

Achieving zero loss and zero duplication requires:

Choosing synchronous send or robust asynchronous callbacks with retry logic.

Enabling synchronous disk flush or quorum‑based replication on the broker.

Acknowledging messages only after durable processing on the consumer side.

Implementing idempotent consumption via database unique keys or external stores such as Redis.