Preventing Message Loss in RabbitMQ and Kafka: Principles, Scenarios, and Solutions

This article is part of a series on MQ interview questions, covering the purpose, advantages, disadvantages, and technology selection of message queues, as well as high‑availability and idempotence considerations.

1. MQ Principles

Data must be neither duplicated nor missing. "No duplication" means messages should not be consumed more than once, which was addressed in the previous section; "no missing" means messages must never be lost, especially when they drive core business logic.

2. Data‑Loss Scenarios

Data loss can occur in two ways: the broker drops the message, or the consumer drops it after receiving it. The following examines RabbitMQ and Kafka separately.

RabbitMQ

A: Producer loses data – During transmission, network issues may cause the producer’s message to be lost before reaching RabbitMQ.

B: RabbitMQ itself loses data – If persistence is not enabled, a broker restart will discard messages. Enabling persistence stores messages on disk, preventing loss except in extremely rare cases where the broker crashes before persisting.

C: Consumer loses data – If a consumer crashes after RabbitMQ marks the message as delivered but before processing it, the message is considered consumed and is lost.

Kafka

A: Producer loses data – Without proper configuration, the producer may drop messages during sending.

B: Kafka loses data – If a broker crashes during leader election and a follower has not yet synchronized, the new leader may miss some data.

C: Consumer loses data – After a consumer receives a message and automatically commits the offset, a crash before processing will cause the message to be considered consumed and thus lost.

3. How to Prevent Message Loss

RabbitMQ

A: Producer loss

1) Use RabbitMQ's transaction feature. The producer starts a transaction, sends the message, and if the broker does not acknowledge receipt, an exception is thrown, the transaction is rolled back, and the message can be resent. If the message is received, the transaction is committed.

channel.txSelect(); // start transaction

try {

// send message

} catch (Exception e) {

channel.txRollback(); // rollback

// retry sending

}

Drawback: Transactions are synchronous and block the producer, reducing throughput.

2) Enable confirm mode. After enabling, each sent message gets a unique ID; RabbitMQ returns an ack if the message is stored successfully or a nack otherwise, allowing the producer to retry.

// enable confirm

channel.confirm();

public void ack(String messageId) { /* handle success */ }

public void nack(String messageId) { /* retry */ }

Confirm mode is asynchronous, allowing the producer to continue sending while awaiting callbacks.

B: RabbitMQ itself loses data – Enable persistence: set the queue to durable and set the message's deliveryMode to 2 (persistent). Both steps are required.

C: Consumer loses data – Disable automatic acknowledgments and manually acknowledge only after processing the message successfully.

Kafka

A: Consumer loss – Disable automatic offset commits and manually commit the offset after processing the message.

B: Broker loss – Configure four parameters:

Set replication.factor > 1 for each topic to ensure at least two replicas.

Set min.insync.replicas > 1 so the leader requires at least one in‑sync follower.

Set producer acks=all so a write is considered successful only after all replicas acknowledge.

Set producer retries=MAX (a very large value) to retry indefinitely on failures.

C: Producer loss – With acks=all, the producer will not consider a message sent until all replicas have stored it; otherwise it will keep retrying, preventing loss.

— THE END —