How to Prevent Message Loss in Kafka: Proven Strategies and Configurations

The primary purpose of introducing an MQ message middleware is system decoupling and traffic control (peak shaving and valley filling).

System decoupling: Using an MQ queue isolates upstream and downstream services, reducing the impact of environment changes.

Traffic control: In high‑concurrency scenarios, MQ smooths traffic spikes and enables asynchronous processing, preventing service crashes.

However, introducing MQ also brings data‑consistency problems.

In a distributed system, when two nodes need to synchronize data, consistency issues arise. The producer sends a message to the MQ , and the consumer must ensure the message is not lost.

When using an MQ queue, three key questions must be addressed:

How to know if a message is lost?

Which stages may cause message loss?

How to ensure messages are not lost?

How to know if a message is lost?

Message loss can be detected through:

Feedback from others: Operations or product managers report missing messages.

Monitoring and alerts: Watch Kafka cluster health, broker status, disk issues, or consumer lag that indicate possible loss.

Case: In a sentiment‑analysis pipeline, if the expected data is absent in Elasticsearch, a new collection task can be issued.

Which stages may cause message loss?

A message passes through three stages: producer, broker, and consumer.

1) Producer side

The Kafka producer buffers messages and sends them in batches via a Sender thread to the broker.

Calling send() does not immediately transmit the message; it is cached and later batched for transmission.

Potential loss scenarios:

Network fluctuations: Unreachable link causes timeout. props.put("retries", "10") can mitigate.

Improper configuration: Missing acks leads to silent failures. Set acks=1 or acks=all and add callbacks.

producer.send(new ProducerRecord<>(topic, messageKey, messageStr),
               new CallBack(){...});

Key acks settings: acks=0: No server acknowledgment; retries are ineffective; high risk of loss. acks=1: Leader writes to its log; if the leader crashes, data may be lost. acks=all (or acks=-1): All replicas must acknowledge; combined with unclean.leader.election.enable=true and proper ISR handling, it ensures durability.

2) Broker side

When a broker receives a batch, it first writes to the OS page cache. The OS periodically flushes this cache to disk asynchronously.

If the broker crashes after the data is in the page cache but before the OS flushes it, power loss can cause message loss. Using a battery‑backed cache mitigates this risk.

Compare MySQL’s “double‑1” strategy (frequent I/O) with Redis’s AOF Everysec strategy, which writes to the buffer every second.

Broker reliability relies on replication (default factor 3):

Leader partition: Handles read/write requests.

Follower partition: Replicates the leader’s data.

Replica synchronization issues can also cause loss; the solution is to use ISR and Epoch mechanisms. ISR (In‑Sync Replicas): When the leader fails, a follower from the ISR becomes the new leader. Epoch: Tracks leader and follower high‑water marks to avoid mismatches (available from Kafka 0.11.x).

Important configuration parameters: acks=-1 or acks=all: All replicas must acknowledge. replication.factor >= 3: Minimum three replicas. min.insync.replicas > 1: At least two replicas must acknowledge (requires acks=-1). unclean.leader.election.enable=false: Prevents non‑ISR followers from becoming leader.

3) Consumer side

Loss scenarios on the consumer side include:

Message backlog: Unconsumed partitions appear as lost messages. Increase consumption speed and process messages in separate threads.

Auto‑commit: Offsets are committed before processing finishes; a crash leads to skipped messages. Disable auto‑commit and commit manually.

Heartbeat timeout: Triggers a rebalance and ejects the consumer from the group. Upgrade the client to version 0.10.2 or later.

Avoid long intervals between two poll calls; tune max.poll.records and max.poll.interval.ms accordingly.

Case study: A data‑sync pipeline performed NLP NER analysis on messages before indexing them into Elasticsearch. An HTTP 500 error in the NER service caused the consumer to time out, and the older client (v0.10.1) coupled heartbeats with poll, leading to rebalance failures.

Resolution:

Set session.timeout.ms=25000 to extend the heartbeat window.

Introduce a circuit‑breaker (Hystrix) to stop consuming when downstream services fail repeatedly.

How to ensure messages are not lost?

Key skills:

Understand every stage from production to consumption.

Monitor Kafka clusters and set up alerts.

Master reliable‑message delivery patterns.

Summary of actions:

Producer: Enable retries ( props.put("retries", "10")), set acks=all, and use callbacks to confirm delivery.

Broker: Use battery‑backed cache, run Kafka ≥ 0.11.x for epoch support, set replication.factor >= 3, configure min.insync.replicas > 1, and disable unclean leader election.

Consumer: Upgrade client to ≥ 0.10.2, disable auto‑commit, commit offsets manually, and increase consumption throughput.