How Kafka Ensures High Availability with Leader‑Follower Replication

In early versions Kafka lacked a high‑availability mechanism, so if a broker failed, all its partitions became unavailable and could cause data loss.

When cluster size grows, the likelihood of failures increases, demanding strong HA.

Starting with version 0.8, Kafka added partition replication. Each partition can have multiple replicas distributed across brokers, ensuring data safety even if a broker crashes.

With replication, Kafka assigns one replica as the Leader and the others as Followers . Producers send messages only to the Leader, and Followers copy the messages from the Leader.

Kafka’s replication model is a hybrid of synchronous and asynchronous approaches. Fully synchronous replication guarantees safety but hurts throughput, while fully asynchronous replication offers high performance but risks data loss.

The Leader maintains an ISR (in‑sync replica) list of Followers that are up‑to‑date. When the Leader receives a new message, it forwards it to the ISR members; once they acknowledge, the Leader confirms the commit to the Producer.

Thus Kafka combines the benefits of both models: some Followers stay in sync, while others replicate asynchronously.

If the Leader fails, a new Leader is elected, preferably from the ISR to preserve data completeness. If no ISR members are available, a non‑ISR replica may be chosen, which can lead to potential data loss.

In extreme cases where all replicas fail, Kafka can either wait for an ISR member to recover (ensuring data reliability but with uncertain recovery time) or immediately promote the first alive replica (restoring availability quickly but possibly with incomplete data). This behavior can be configured to balance availability and consistency.