Why Kafka 2.8’s Raft Integration Eliminates ZooKeeper – Benefits & Caveats

Kafka 2.8.0 has been released, and this version brings a major improvement:

It implements the Raft distributed consensus mechanism, meaning it can run independently without ZooKeeper.

In Kafka, ZooKeeper plays a crucial role by storing Kafka’s metadata, including partition and broker information, and handling the election of the Kafka Controller.

Because ZooKeeper is an external system, deploying a Kafka cluster also requires deploying, managing, and monitoring a ZooKeeper ensemble.

ZooKeeper has its own configuration methods and management tools that differ from Kafka’s, so operating two separate distributed systems increases complexity and makes problems more likely. For example, enabling certain security features now requires configuration in both Kafka and ZooKeeper.

Beyond complexity, external storage also reduces system efficiency . For instance, each time a Kafka cluster starts, the Controller must load cluster state from ZooKeeper. After a new Controller is elected, it must also load potentially large amounts of metadata, which can become a performance bottleneck.

These complexity and efficiency issues have long been pain points for Kafka users, and the Kafka team has been working to remove the dependency on ZooKeeper. Kafka 2.8.0 finally achieves this.

With Raft mode, metadata and configuration are stored in the @metadata topic, which is automatically replicated across the cluster, making Kafka much simpler and lighter.

However, it is important to note that ZooKeeper‑less Kafka is still an early‑stage feature and is not yet mature, so it should not be used in production environments.