Zookeeper in Distributed Systems: Roles in Kafka, Hadoop, HBase, and Solr

Zookeeper is a high‑performance, highly reliable distributed coordination system, an open‑source implementation of Google’s Chubby, offering consistency services like configuration maintenance, naming, distributed synchronization, and group services.

It uses the ZAB (Zookeeper Atomic Broadcast) protocol, a classic implementation of the Paxos consensus algorithm, to propagate leader state changes and ensure consistency between leader and followers.

Typical Zookeeper use cases include data publish/subscribe (configuration centers), global ID naming services, distributed coordination (watchers, asynchronous notifications), heartbeat detection, task progress reporting, master election, and distributed locks.

In Kafka, Zookeeper stores metadata such as broker registrations, topic and partition information, elects partition leaders, and (in older versions) holds consumer offset data.

Within Hadoop, Zookeeper provides HA for HDFS NameNode and YARN ResourceManager; the article shows the YARN RM HA architecture.

In HBase, Zookeeper coordinates the cluster by maintaining session information for HMaster and RegionServers, tracking healthy servers, and handling failover notifications.

SolrCloud relies on Zookeeper for configuration management, cluster state, and leader election, whereas Elasticsearch uses its own built‑in coordination mechanism.

The article concludes that Zookeeper is widely used across many distributed systems beyond the examples given, including Spring Cloud and Dubbo micro‑service architectures.