Understanding ZooKeeper: Core Concepts, Architecture, and Practical Insights

ZooKeeper is an open‑source, high‑availability coordination service widely used in large distributed systems. It offers simple primitives for tasks such as leader election, group membership, and metadata management, allowing developers to focus on application logic rather than coordination details.

Purpose and Audience

The article targets developers new to distributed systems and operations engineers managing ZooKeeper clusters, aiming to explain basic concepts, source‑code insights, and practical experience for large‑scale deployments.

ZooKeeper Overview

Inspired by file‑system APIs, ZooKeeper presents a hierarchical tree of znodes with a simple API. It runs on Java with bindings for Java and C, exposing primitives for synchronization, configuration maintenance, and naming.

Data Structures

ZooKeeper stores data in a tree‑like structure of znodes. Each znode can be one of four types:

Persistent

Ephemeral

Persistent Sequential

Ephemeral Sequential

Persistent znodes remain after the creator disconnects, while ephemeral znodes are removed when the session ends. Sequential znodes receive a monotonically increasing integer suffix.

API Commands

create /path data          # create a znode with data</code>
<code>delete /path              # delete a znode</code>
<code>exists /path              # check if a znode exists</code>
<code>setData /path data       # set data of a znode</code>
<code>getData /path            # retrieve data of a znode</code>
<code>getChildren /path        # list children of a znode

All read/write operations replace or return the entire znode content; partial reads/writes are not supported.

Watch and Notification

Clients register one‑time watches on znodes. When the watched znode changes, the server sends a single notification. Watches exist only in memory and are cleared when the client disconnects.

Architecture

ZooKeeper clusters operate in either standalone or quorum mode. In quorum mode, a set of servers (the ensemble) elect a leader that serializes all state‑changing requests. Followers replicate the leader’s proposals, and optional observer nodes receive updates without participating in elections, improving read scalability.

Leader Election

Each server starts in LOOKING state, exchanging votes containing its server ID (sid) and the latest transaction ID (zxid). The server with the highest zxid (and highest sid if zxids tie) becomes the leader. Once a majority acknowledges the leader, the election completes.

ZAB Protocol (ZooKeeper Atomic Broadcast)

Leader sends a PROPOSAL to all followers.

Followers respond with an ACK upon receiving the proposal.

When a quorum of ACKs is collected, the leader broadcasts a COMMIT to finalize the transaction.

This two‑phase commit ensures total order and atomicity of state updates across the ensemble.

Server Components

Requests flow through a pipeline of processors:

PrepRequestProcessor : validates client requests and creates transactions for write operations.

SyncRequestProcessor : persists transactions to the transaction log and creates snapshots.

FinalRequestProcessor : applies transactions to the data tree or reads data for read‑only requests.

In quorum mode, additional processors such as ProposalRequestProcessor , AckRequestProcessor , and CommitRequestProcessor handle proposal distribution, acknowledgment, and commit phases.

Storage: Logs and Snapshots

Each server writes transactions to a sequential log on disk. To guarantee durability, the log must be flushed before acknowledging the client. Group commits batch multiple transactions into a single disk write, and pre‑allocation (padding) reduces fragmentation. Periodically, servers take snapshots of the entire data tree, which can be taken without pausing request processing, though snapshots may be slightly stale.

Sessions

Sessions uniquely identify client connections and are tracked by the leader in quorum mode (or by the single server in standalone mode). Heartbeats—either explicit ping messages or any client request—refresh the session timeout, ensuring temporary znodes and watches remain valid.

Client Library

The primary client classes are ZooKeeper and ClientCnxn. ZooKeeper establishes a session, while ClientCnxn manages socket connections, server list rotation, and watch re‑registration after reconnection.

Serialization

ZooKeeper uses Hadoop’s Jute library for serializing messages and transactions for network transmission and disk storage.

Use Cases

Major projects such as Apache HBase, Apache Kafka, and Apache Solr rely on ZooKeeper for leader election, metadata storage, and coordination, earning it the nickname “The King of Coordination for Big Data.”

References

ZooKeeper election analysis: https://juejin.im/post/5cc2af405188252da4250047

Apache ZooKeeper official site: https://zookeeper.apache.org/

ZooKeeper GitHub repository: https://github.com/apache/zookeeper

“ZooKeeper: Distributed Process Coordination” (Flavio Junqueira, et al.)

Various ZooKeeper source‑code analysis blogs