Why Does One Kafka Broker Failure Halt All Consumers? HA & Replication Explained

1. Kafka Outage and HA Issue

The discussion starts with a Kafka broker crash that caused consumers to stop receiving messages, highlighting a puzzling high‑availability problem despite having three broker nodes.

2. Kafka Multi‑Replica Redundancy Design

Key concepts of Kafka are introduced:

Broker (node) : a Kafka server instance.

Topic : logical grouping of messages; producers and consumers use the topic name to write and read.

Partition : a topic is split into one or more partitions; each partition belongs to a single broker.

Offset : the position of a message within a partition, used by consumers to track progress.

3. Replication Mechanism

Since version 0.8 Kafka introduced replica sets for each partition. Each replica set contains one Leader and multiple Followers. Producers and consumers interact only with the Leader; Followers pull data from the Leader to stay synchronized. Kafka maintains an In‑Sync Replica (ISR) list of Followers that are sufficiently up‑to‑date. If a follower falls behind, it is removed from the ISR.

4. Ack Parameter and Reliability

The producer’s request.required.acks setting determines how many acknowledgments are required for a send to be considered successful:

0 – fire‑and‑forget; messages may be lost.

1 – only the Leader’s receipt is required; if the Leader fails before Followers sync, data can be lost.

All (or -1) – the Leader and all ISR Followers must acknowledge; this provides the strongest durability, but still requires at least two in‑sync replicas to avoid loss.

5. Root Cause: __consumer_offset Topic

The built‑in __consumer_offset topic stores consumer offsets. By default it has 50 partitions with a replication factor of 1, creating a single point of failure. When the broker holding these partitions crashes, all consumers stop processing.

6. Solution

Delete the existing __consumer_offset topic (e.g., by removing its log files) and let Kafka recreate it.

Set offsets.topic.replication.factor to 3 so the offset topic also has three replicas, matching the number of brokers.

7. Conclusion

When the replication factor of both regular topics and the __consumer_offset topic matches the number of brokers, Kafka’s multi‑replica design ensures high availability, and the cluster remains operational even after a broker failure.