Detecting and Preventing Message Loss in Kafka Message Queues

The article introduces the primary goals of using a message queue (MQ) such as Kafka: system decoupling and traffic shaping (peak‑shaving). While MQ solves these problems, it also brings data‑consistency challenges because messages must not be lost between producer and consumer.

Key Questions When Using MQ

How to know if a message has been lost?

Which stages of the pipeline may cause loss?

How to guarantee that messages are not lost?

How to Detect Message Loss

Two main sources of detection are:

User feedback: product managers (PM) or operations report missing messages.

Monitoring alerts: Kafka cluster anomalies (broker crash, disk issues, consumer lag) trigger alarms that indicate possible loss.

Typical Scenarios and Diagnosis

Producer side problems:

Network interruptions cause send time‑outs. props.put("retries", "10") can mitigate.

Improper acks configuration. Use acks=1 or acks=all and enable callbacks.

producer.send(new ProducerRecord<>(topic, messageKey, messageStr),
               new CallBack(){...});

Important acks values: acks=0: fire‑and‑forget, high loss risk. acks=1: leader write acknowledgment only. acks=all (or -1): all replicas must acknowledge.

Broker side issues:

Messages reside first in OS page cache; a broker crash after caching but before flush can cause loss.

Power loss leads to loss of in‑memory data. Use battery‑backed cache to protect.

Kafka log‑flush configuration (default is OS‑driven):

# Recommended defaults – let OS decide when to flush
log.flush.interval.messages=10000   # flush after 10k messages
log.flush.interval.ms=1000        # flush every second
flush.messages.flush.ms=1000     # topic‑level flush interval
flush.messages=1                 # flush every message

Replication ensures reliability. Typical settings:

replication.factor=3

min.insync.replicas>1

(requires acks=-1) unclean.leader.election.enable=false to avoid electing out‑of‑sync followers.

ISR (In‑Sync Replicas) and epoch mechanisms prevent data loss when the leader fails.

Consumer side pitfalls

Message backlog makes it appear as loss – increase consumption speed or parallelize processing.

Automatic offset commits can acknowledge before processing finishes. Disable with auto.commit=false and commit manually.

Heartbeat timeout triggers a rebalance, ejecting the consumer. Upgrade client to ≥ 0.10.2 and tune session.timeout.ms, max.poll.records, and max.poll.interval.ms.

Practical Checklist to Ensure No Message Loss

Producer : enable retries, set acks=all, use callbacks.

Broker : use battery‑backed cache, run Kafka ≥ 0.11.x (epoch support), set replication factor ≥ 3, configure min.insync.replicas and disable unclean leader election.

Consumer : upgrade client version, disable auto‑commit, increase poll frequency, adjust session and poll timeouts, consider circuit‑breaker (Hystrix) for downstream service failures.

By mastering these configurations, monitoring Kafka clusters, and understanding the end‑to‑end message flow, engineers can reliably prevent message loss in distributed systems.