Kafka 2.8 Introduces KRaft: Running Without ZooKeeper

In the upcoming Kafka 2.8 release, the distributed publish‑subscribe system adopts an internal Quorum controller (KRaft) to replace ZooKeeper, enabling users to run Kafka without any ZooKeeper dependency. This reduces computational resources, improves performance, and allows larger cluster deployments.

Historically, Apache ZooKeeper served as the coordination service for Kafka and similar distributed systems. All broker nodes registered with ZooKeeper at startup, and ZooKeeper stored metadata about broker status changes. While powerful, ZooKeeper added complexity as a separate component, prompting the shift to an internal controller.

The migration effort began in April of the previous year, and by version 2.8 users can operate Kafka in KRaft mode. In this mode, metadata previously managed by both the Kafka controller and ZooKeeper is consolidated into the new Quorum controller, which runs inside the Kafka cluster. Users with special requirements can still deploy the Quorum controller on dedicated hardware.

KRaft uses an event‑driven mechanism to track cluster metadata, replacing many RPC‑based tasks with event‑driven log replication. This change enhances Kafka’s ability to support more partitions.

Removing ZooKeeper makes Kafka lighter and more suitable for small‑scale workloads, edge deployments, and lightweight hardware solutions, addressing the perception of Kafka as a heavyweight infrastructure.

However, the early‑access version lacks support for certain features such as ACLs, security, transactions, partition reassignment, and JBOD. The Kafka team plans to add these in later releases, and they advise against using KRaft in production until the features mature.

References: https://www.confluent.io/blog/kafka-without-zookeeper-a-sneak-peek/ https://www.ithome.com.tw/news/143569