How Zookeeper Powers Kafka: Key Roles Explained

Zookeeper is an inseparable component of Kafka, handling critical coordination tasks for brokers and consumers.

1. Broker

Status : Zookeeper records the alive status of all brokers; brokers send heartbeat requests to report their state and Zookeeper maintains a list of running brokers.

Controller election : Among multiple brokers, one is elected as the controller, which manages partition and replica states; Zookeeper performs this election.

Quota and ACL : Client production and consumption quotas, as well as topic access control information, are stored in Zookeeper.

ISR tracking : Zookeeper keeps real‑time information about the in‑sync replica (ISR) set; only replicas in ISR are eligible to become leaders.

Node and topic registration : Zookeeper stores registration data for nodes and topics as temporary nodes, which disappear when the session ends.

Topic configuration : Topic metadata such as partition count and replica placement are persisted in Zookeeper.

2. Consumer

Offset : In older Kafka versions, consumer offsets were stored in Zookeeper; newer versions use Kafka’s own offset manager.

Registration : Consumers register themselves by creating temporary nodes; they are automatically removed when the consumer goes down.

Partition registration : Each partition is consumed by only one consumer in a group; Zookeeper tracks the relationship between partitions and consumers.

Recommended reading: Interface‑level fault handling strategies, using Canal for data heterogeneity, distributed transaction TCC, determining element existence in massive datasets, why Kafka uses consumer groups.