Inside ZooKeeper: Source Code Walkthrough, Thread Model, and Real‑World Ops Tips

Introduction

ZooKeeper is an open‑source coordination service that offers high availability, strong consistency, and high performance for large distributed systems. It is widely used as a learning platform for distributed components and as a production‑grade service.

Purpose and Audience

The article targets developers new to distributed systems and operations engineers responsible for ZooKeeper deployments, aiming to explain core concepts, source‑code structure, and practical experience.

Client Thread Model

The client entry point is ZooKeeperMain. Its run() method creates a console reader and enters a loop waiting for user input while establishing a session via connectToZK. The ZooKeeper constructor creates a NIO socket and starts two daemon threads: sendThread (sends heartbeats and user commands) and eventThread (processes queued events and watches). A ClientCnxn instance, implemented by ClientCnxnSocketNIO, handles I/O.

Client Source‑Code Highlights

Key data structures in ClientCnxn.java:

/**
 * Packets that have been sent and are awaiting a response.
 */
private final LinkedList<Packet> pendingQueue = new LinkedList<Packet>();

/**
 * Packets that need to be sent.
 */
private final LinkedBlockingDeque<Packet> outgoingQueue = new LinkedBlockingDeque<Packet>();

When a request is submitted via cnxn.submitRequest(), it is wrapped in a Packet and placed into outgoingQueue. The sendThread consumes the queue, writes data to the server, and blocks the calling thread until the packet’s finished flag is set by the response handling logic.

Server Thread Model

The server side uses NIOServerCnxnFactory, which starts three main threads:

AcceptThread : accepts client connections and hands them to a selector.

SelectorThread : processes read/write events for each channel.

ConnectionExpirerThread : checks for session timeouts.

When a client connects, the server creates a ServerCnxn object that represents the session and implements the Watcher interface.

Watch Mechanism

ZooKeeper supports watches on znode data, children, creation, and deletion. The server maintains a WatchManager that stores in‑memory watch registrations. When a watch triggers, the server serializes a WatchEvent and sends it to the client, where the eventThread deserializes it and invokes the user‑provided Watcher.process() implementation.

GetData Example (Synchronous)

Client side:

Create a WatchRegistration (e.g., DataWatchRegistration) for the target path.

Build a request with type GetData and set the watch flag.

Call ClientCnxn.submitRequest(), which enqueues the request.

Server side:

The FinalRequestProcessor handles the request by calling zks.getZKDatabase().getData(path, stat, watch ? cnxn : null). If a watch is present, the server stores the watch in a path‑to‑watch map.

Operational Experience

The authors manage a ZooKeeper deployment with dozens of clusters across multiple regions (e.g., Shanghai 168 nodes, Guangzhou 95 nodes, Beijing 41 nodes). The total number of znodes exceeds 30 million, and watches can reach 160 million, causing significant scaling challenges.

Disaster‑Recovery Cluster

Because leader election can stall and older versions lack dynamic reconfiguration, a standby cluster is built by copying transaction logs and snapshots. Manual switchover reduces downtime but still requires careful coordination.

Observer Load Issue

During burst writes, large sub‑trees cause observers to experience high CPU while serializing getChildren results, leading to delayed configuration propagation. Mitigations include scaling observers, hierarchical node grouping, and multi‑process clients.

Full GC‑Induced Session Failures

Long GC pauses on server nodes cause session timeouts, forcing clients to reconnect and generating a connection‑spike that can overwhelm the cluster.

Network Partition Impact

When a region loses connectivity, observers are expelled, clients disconnect, and after recovery a massive reconnection surge can overload observers, especially if many connections remain in CLOSE_WAIT state.

initLimit and syncLimit Tuning

initLimit defines the timeout for a follower’s initial sync with the leader (in ticks). Too low a value prevents slow followers or observers from joining, especially with large snapshots or slow networks.

syncLimit controls the timeout for ongoing sync operations. If a follower falls too far behind, the leader may drop it, causing loss of quorum.

Optimization Measures

Monitor and expand observer capacity; track CPU and memory.

Group large child nodes to reduce getChildren payload.

Run multiple client processes with balanced node assignments.

Introduce a hold‑time before pulling data after a watch to batch updates.

Limit connection spikes during failures by capping established connections and cleaning CLOSE_WAIT sockets.

Adjust initLimit and syncLimit based on network latency and data size.

References

ZooKeeper election analysis – https://juejin.im/post/5cc2af405188252da4250047

Apache ZooKeeper official site – https://zookeeper.apache.org/

ZooKeeper GitHub repository – https://github.com/apache/zookeeper

“ZooKeeper – Distributed Process Coordination” (Flavio Junqueira, Reed)