How Kafka Uses ZooKeeper for Metadata Management and Client Coordination

Kafka and ZooKeeper Overview

Kafka uses ZooKeeper to store essential cluster metadata, similar to RocketMQ's NameServer. The article outlines what information is kept in ZooKeeper and how both clients and brokers interact with it.

ZooKeeper Node Structure

ids subtree (ephemeral nodes) : Stores broker information such as address, version, and start time. Each online broker creates an ephemeral node under /brokers/ids/[brokerId]. If a broker crashes, its node disappears.

topics subtree : Each child node under /brokers/topics represents a topic name.

partitions subtree : Under each topic node there is a fixed partitions node; its children are the partition numbers.

state (ephemeral) nodes : Under each partition node, a state node holds the current leader broker ID and the list of ISR broker IDs. It is created by the leader broker and vanishes if the leader fails.

How a Kafka Client Finds a Broker

The client first reads the state temporary node of the desired partition to obtain the leader broker ID, then looks up the corresponding broker information in the ids subtree to get the broker’s host and port.

Clients do not connect directly to ZooKeeper. Instead, they request metadata from a broker via RPC, cache it locally, and use it for producing and consuming.

Broker Handling of Metadata Requests

All RPC requests enter the broker through KafkaApis#handleTopicMetadataRequest. The method extracts the requested topics, filters out unauthorized topics, gathers topic metadata from the cache, adds the list of alive brokers, builds a response, and sends it back to the client.

def handleTopicMetadataRequest(request: RequestChannel.Request): Unit = {
    // Determine which topics are requested
    val topics = if (metadataRequest.isAllTopics) metadataCache.getAllTopics()
    // Filter out topics the client is not authorized to describe
    // Retrieve topic metadata from the cache
    // Get the list of alive brokers (ids subtree)
    // Build and send the response
}

Metadata Request Construction Details

When a client needs fresh metadata, the broker creates a MetadataRequest.Builder rather than a full MetadataRequest. The builder is later converted to a real request just before sending. The request is queued and sent asynchronously.

The concrete data of the request is encapsulated in the MetadataRequestData class.

Best Practices and Future Directions

Kafka’s availability is tightly coupled to ZooKeeper; if the ZooKeeper ensemble fails, the entire Kafka cluster becomes unavailable. Developers are discussing a new metadata service to replace ZooKeeper.

For large‑scale deployments, it is recommended to split the cluster into multiple independent small clusters, each with its own ZooKeeper ensemble. This reduces the impact of a ZooKeeper outage to only the affected small cluster.

Reference: https://www.bcoder.top/2019/12/14/%E6%B6%88%E6%81%AF%E9%98%9F%E5%88%97%E4%B9%8BKafka%E5%8D%8F%E8%B0%83%E6%9C%8D%E5%8A%A1ZooKeeper/