How to Prevent Message Loss in Kafka: Practical Tips and Configurations

Introducing a message queue (MQ) primarily aims to decouple systems and smooth traffic spikes, but it also brings data consistency challenges that require careful handling.

Why Message Loss Matters

In distributed systems, data synchronization between nodes can cause consistency issues; messages must not be lost from producer to consumer.

Three Critical Questions

How to detect message loss?

Which stages may lose messages?

How to guarantee no loss?

1. Producer Side Issues

Kafka producers buffer messages and send them in batches via a Sender thread. Loss can happen due to:

Network instability: timeouts or unreachable broker cause messages to stay unsent. Solution: set retries=10.

Improper configuration: no ACKs, no callbacks, no logging. Solution: set acks=1 or acks=all and add a callback, e.g.

producer.send(new ProducerRecord<>(topic, key, value), new CallBack(){...});

.

Key ACK settings: acks=0: fire‑and‑forget, high throughput but data may be lost. acks=1: leader writes to its log, moderate latency, risk if leader fails. acks=all (or -1): wait for all in‑sync replicas, safest when combined with unclean.leader.election.enable=false.

2. Broker Side Issues

Kafka brokers write batches to the OS page cache first, then flush to disk asynchronously. Risks:

If the broker crashes after writing to cache but before flush, data remains safe because the cache is still in memory.

If the machine loses power, data in RAM is lost.

Mitigations include using battery‑backed cache and configuring replication:

Set replication.factor>=3 (minimum three replicas).

Set min.insync.replicas>1 so a message is considered committed only after reaching at least two replicas (requires acks=all).

Disable unclean leader election: unclean.leader.election.enable=false.

Enable epoch mechanism (Kafka 0.11+).

3. Consumer Side Issues

Message loss can also stem from the consumer:

Message backlog: unprocessed partitions appear as lost. Solution: increase consumption speed, process records in separate threads.

Auto‑commit: offsets are committed before processing finishes; a crash leads to skipped messages. Solution: set auto.commit=false and commit manually after processing.

Heartbeat timeout / rebalance: slow consumers are evicted from the group. Adjust max.poll.records and max.poll.interval.ms accordingly, and upgrade client to >=0.10.2 where heartbeat is decoupled from poll().

Operational Tools

Useful Kafka commands:

# Get message count for a topic
./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic test_topic

# List consumer groups
./kafka-consumer-groups.sh --list --bootstrap-server 192.168.88.108:9092

# Describe consumer group offsets
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group console-consumer-1152 --describe

Monitoring alerts (e.g., broker down, disk issues, consumer lag) are essential to detect loss early.

Case Study: NLP‑Driven Data Sync

A pipeline pulls messages from Kafka, runs NER analysis, and writes results to Elasticsearch. After a period, consumption stopped due to frequent rebalances and HTTP 500 errors from the NER service. The root cause was an outdated client (v0.10.1) lacking separate heartbeat threads and an unstable NER service.

Fixes applied:

Increase session.timeout.ms to 25 s.

Introduce a circuit‑breaker (Hystrix) to stop consuming when downstream services fail repeatedly.

Upgrade client to >=0.10.2.

Summary Checklist

Understand the full message flow: producer → broker → consumer.

Monitor Kafka cluster health and consumer lag.

Configure reliable delivery: acks=all, retries, callbacks.

Ensure broker durability: replication factor ≥3, min.insync.replicas>1, disable unclean leader election, use battery‑backed cache.

Prevent consumer loss: disable auto‑commit, tune poll settings, upgrade client version.