How to Prevent Kafka Message Loss: Producer, Broker, and Consumer Settings

Producer: Preventing Message Send Loss

The main cause of producer‑side loss is network jitter or broker failure, which can cause send failures. Configure the producer to use full acknowledgments and enable idempotence so that messages are reliably written to the broker. acks = all – the producer waits for all in‑sync replicas (ISR) to confirm the write before considering it successful. This is more reliable than the default acks = 1 but adds latency. enable.idempotence = true – activates an idempotent producer (available since Kafka 0.11). It uses a Producer ID (PID) and sequence numbers to avoid duplicate sends.

Broker: Preventing Storage Loss

Broker‑side loss can stem from disk failures or unclean leader elections. High‑availability settings ensure that messages are persisted and replicated across multiple brokers. replication.factor = 3 (or higher) – each partition has at least three replicas. min.insync.replicas = 2 – when the producer sends a message, at least two replicas must acknowledge the write (used together with acks = all). unclean.leader.election.enable = false – disables unclean leader election, preventing a non‑ISR replica from becoming leader and causing data loss. log.retention.hours = 168 – extends log retention to 7 days, ensuring messages are not deleted prematurely.

Consumer: Preventing Consumption Loss

Consumer loss typically occurs when offsets roll back after a restart or when the process crashes. Use manual offset commits and external persistence to keep consumption progress reliable. enable.auto.commit = false – disables automatic offset commits. isolation.level = read_committed – enables transactional reads, ensuring only committed messages are consumed (exactly‑once semantics). auto.offset.reset = earliest – on first start, the consumer reads from the earliest offset, avoiding missed historical messages.