ZooKeeper & ZAB Protocol Deep Dive: Leader Election, Crash Recovery, and Write Path Internals
This comprehensive analysis dissects ZooKeeper's architecture, the ZAB consensus protocol, leader election mechanics, write request lifecycle, crash recovery guarantees, and operational pitfalls through real-world failure scenarios and source code walkthroughs.
Three Failure Incidents and Their Lessons
The article opens with three real-world incidents that illustrate ZooKeeper's counter-intuitive behaviors:
Election "fake death" : A five-node cluster experiences network packet loss on ports 2888/3888. The Leader fails to collect enough ACKs, followers transition to LOOKING state, and the entire cluster becomes unavailable for writes for 30 seconds — despite all five processes being alive with low CPU and no ERROR logs. Lesson: Availability is determined by quorum, not process liveness.
Full GC session avalanche : A JVM Full GC pauses for 8 seconds while sessionTimeout is 10 seconds. The client cannot send heartbeats; the server's SessionTracker expires the session, deleting 3,000 ephemeral nodes simultaneously. Gateways watch these nodes and remove the instance, shifting traffic to two remaining nodes which then also time out. Lesson: Session expiry uses server-side clocks; client STW does not pause server time.
Disk-full "half-available" state : txnlog fills the system disk, one node exits quorum (5→4). An admin restarts that node, losing its data (5→3). A network partition then isolates one node (2 vs 1). The two-node side has quorum=2 and continues writes; the single node rejects writes. Lesson: Quorum is the sole write switch; reducing nodes directly cuts write capacity.
Seven-Piece Puzzle of a CP System
2.1 Data Model: znode Tree and Four Node Types
ZooKeeper's in-memory model is a filesystem-like tree where each node (znode) can both store data (default 1MB limit, recommended <1KB) and have children — it is both directory and file . This naturally expresses "group + member" relationships (parent = service name, children = instances).
Four node types from two binary attributes (persistent/ephemeral × sequential/non-sequential):
PERSISTENT : Exists until explicit delete. Used for config, metadata, lock parent paths.
EPHEMERAL : Bound to session lifecycle; auto-deleted on session end (close or timeout). Powers service registration, distributed locks, leader election — "process dies, node disappears" at zero code cost.
PERSISTENT_SEQUENTIAL : Server appends a 10-digit monotonic sequence number (e.g., lock-0000000007). Suits global queueing semantics.
EPHEMERAL_SEQUENTIAL : Intersection of above; preferred for fair locks and leader election . Each contender creates one; smallest sequence wins; watch predecessor for release.
Critical constraint : Ephemeral nodes cannot have children. This avoids cascading subtree deletion on session expiry (a write operation requiring synchronous replication). Engineering corollary: instance nodes must be leaves — do not hang health-status children under /services/order/instance-1.
2.2 Three Roles and Three States
Each server has a fixed myid (1–255 in dataDir/myid) but assumes one of three runtime roles:
Leader : Unique; handles all writes and proposes transactions.
Follower : Participates in voting and write acknowledgments; serves reads.
Observer : Syncs data, serves reads, does not vote nor count toward quorum — scales read throughput without impacting write latency.
Corresponding server states (ZAB's core abstraction):
LOOKING : Searching for Leader; provides no service.
FOLLOWING : Confirmed Leader; acting as follower.
LEADING : Self is Leader.
States transition: Leader failure → Follower returns to LOOKING; election win → LOOKING to LEADING. Incident 1's "fake death" is processes alive but stuck in LOOKING.
Practical health check : Don't check process; check state and recognized Leader via four-letter command echo stat | nc ip 2181 (shows Mode: leader/follower). If stat times out, the node cannot reach quorum and is spinning in LOOKING.
2.3 Write Path: Why All Writes Must Go Through Leader
Read/write asymmetry:
Reads execute locally on any node (enabling 100k+ read throughput).
Writes must be proposed by Leader and acknowledged by quorum (write throughput typically 10–20k, heavily dependent on disk fsync).
Consequence: Reads may return stale data . Follower data lags Leader; reads return locally — ZK guarantees only sequential consistency (same client's requests execute in order), not "read latest at any moment". To close the window, explicitly call sync() to align Follower with Leader before reading. Distributed lock logic that reads lock node without sync could theoretically see "lock not yet created" — rare in practice but a frequent interview question.
Why Leader-only writes? To assign a globally unique total order to all writes . If any node could write, concurrent "create /lock" requests would need conflict resolution. Funneling all writes to one Leader reduces ordering to a queueing problem: Leader assigns sequence numbers (zxid) by arrival order. ZAB's essence is making this numbering process a state-machine replication protocol that survives crashes and partitions.
2.4 ZAB's Two Phases: Crash Recovery and Message Broadcast
ZAB has only two modes; understanding both explains ZK's consistency:
Message Broadcast (stable, Leader exists) : Leader wraps each write into a proposal with zxid, broadcasts via FIFO queues to all Followers. Followers persist to txnlog and ACK. Leader collects quorum ACKs (including self), broadcasts COMMIT. All nodes apply to in-memory DataTree. Leader responds to client. This is a two-phase commit variant without the prepare phase — FIFO ordering eliminates the need to ask "can commit?" then "commit."
Crash Recovery (Leader failure or startup) : Three sub-phases:
Leader Election : Elect node with highest zxid.
Discovery : New Leader gathers Followers' zxids to determine sync baseline.
Synchronization : Bring Followers up to date (catch up or truncate) to match Leader. Only after sync completes does broadcast mode begin and service resume.
Recovery must guarantee two properties (source of all ZAB complexity):
Committed transactions never lost → New Leader must have max zxid, ensuring it holds all committed transactions.
Proposed-but-uncommitted transactions must be discarded → New Leader forces Followers to truncate entries beyond its own zxid.
Together: "Committed never lost, uncommitted never applied."
2.5 zxid: 64 Bits Encoding Time and Order
zxid is a 64-bit long split into:
High 32 bits = epoch (term number), increments on each leadership change.
Low 32 bits = counter , increments per write within a single Leader term.
This structure elegantly encodes both "time" and "order":
Compare new vs old : Compare epoch first, then counter → natural total order. Election "who has larger zxid" uses this total order; no extra clocks needed.
Identify era : txnlog files named by zxid (e.g., log.0000000005 where 5 is epoch). Recovery instantly sees which logs belong to previous term vs current.
Prevent split-brain : Old Leader restarts with old epoch; new Leader has higher epoch. Old Leader receives NEWLEADER message and must step down to FOLLOWING. Epoch proves "whose era it is" — more reliable than process liveness.
Counter overflow? 32-bit unsigned ≈ 4.3B writes. At 20k TPS that's <3 days — but each leadership change increments epoch and resets counter. Practical trigger: single term writing 2^31 triggers term-change protection . Hence ZK's recommendation for periodic rolling restarts (which induce term changes), explaining some "ran six months then sudden anomaly" mysteries.
2.6 Voting: Who Becomes Leader
Since ZK 3.4, default is FastLeaderElection . In LOOKING state, each node broadcasts a vote tuple: (logicClock, state, myid, zxid) — logical clock, own state, own id, highest zxid seen.
Comparison rules (priority order) :
Higher zxid wins.
If zxid equal, higher myid wins.
"Compare zxid first" ensures new Leader has freshest data (recovery guarantee 1). "Compare myid on tie" merely breaks symmetry to make election result deterministic and reproducible — otherwise equal-data nodes could flip-flop across rounds. Quorum of identical votes elects; otherwise update vote and repeat until convergence.
Common misconception : ZK election is not "first come first served" but "most up-to-date data wins". Thus after cluster restart, the Leader is often not the highest myid but the node with most complete data. Operational tip: rolling restart Followers first, Leader last, significantly reduces term changes and recovery time.
2.7 Quorum and Observer: Availability Arithmetic
Quorum = majority = N/2 + 1 voting nodes. Fault tolerance derived:
3 nodes tolerate 1 failure
5 nodes tolerate 2 failures
7 nodes tolerate 3 failures
Note: "tolerate" ≠ "can stop for maintenance". Rolling restart stops one node at a time; a 3-node cluster is already at the edge — one maintenance + one unexpected failure loses write capability.
Why production recommends 5 over 3? Not performance — maintenance window : 5 nodes allow one planned maintenance + one unexpected failure. Why rarely 7? Write latency grows with node count (more ACKs, more network round-trips) while fault-tolerance gains diminish; 5 is the latency-reliability balance point .
Observer breaks the "read scaling requires adding voting nodes" dilemma: Syncs all data, serves reads, but does not vote nor count toward quorum . Add ten Observers for read throughput; write latency unchanged. Trade-off: Observer data may be staler; cannot become Leader. Ideal for read-heavy registry scenarios.
Core Flow: Complete Journey of One Write Request
Combining sections 2.3 and 2.4, a create /lock timeline (client connected to Follower F1, 5-node cluster):
T+0 Client sends create /lock to F1
T+1 F1 forwards REQUEST to Leader
T+2 Leader assigns zxid, wraps proposal
T+3 Leader broadcasts PROPOSAL to all
T+4 Followers write txnlog and fsync
T+5 Followers send ACK to Leader
T+6 Leader receives 3/5, decides commit
T+7 Leader broadcasts COMMIT to all
T+8 All nodes apply to DataTree (memory)
T+9 Leader sends RESPONSE to F1
T+10 F1 returns ok to clientKey point 1 : Follower doesn't just forward and wait; it also participates. Leader's PROPOSAL broadcasts back to F1 — F1 must also persist and ACK. So "5-node quorum" = Leader + any two others, not "Leader + three Followers".
Key point 2 : ACK only after fsync . Follower writes txnlog, forces to disk, then ACKs. This is the physical guarantee of "committed never lost" — if ACK after page cache only, a power loss loses the log while the transaction is considered committed, causing data loss on recovery. This fsync + one network RTT is ZK's write latency floor.
Key point 3 : COMMIT is unidirectional from Leader. Followers never judge "is quorum reached?"; they only await Leader's COMMIT. Decision authority centralized in Leader avoids divergent commit decisions — but if Leader "fake dies" (can send PROPOSAL but not COMMIT), writes stall in intermediate state: all nodes persisted, none dare apply.
Key point 4 : In-memory application is replayable. T+8 applies to DataTree in memory; txnlog already on disk. On restart, node loads latest snapshot then replays subsequent txnlog to reconstruct pre-crash state. Snapshot = "compressed history", txnlog = "incremental" — universal pattern in state-machine replication (Redis RDB+AOF, MySQL full backup+binlog).
Core Source Code: Five Sections to Understand State Machine and Replication
4.1 QuorumPeer.run: Scheduler for Three States
The entire ZK server "heartbeat" is this loop: check current state, execute corresponding branch, re-evaluate after branch returns. Election failure → re-elect; Leader failure → back to LOOKING. Incident 1's "stuck in LOOKING" is this loop spinning without convergence.
// zookeeper/server/quorum/QuorumPeer.java
while (running) {
switch (getPeerState()) {
case LOOKING:
// re-elect if no result
setCurrentVote(makeLEStrategy().lookForLeader());
break;
case FOLLOWING:
setFollower(makeFollower(zk));
follower.followLeader();
break;
case LEADING:
setLeader(makeLeader(zk));
leader.lead();
break;
}
}Note followLeader() and lead() are blocking: as long as role unchanged, they stay in that method working ; on Leader loss or self-demotion, method returns and loop re-evaluates state. This "blocking role loop" is easier for state consistency than event-driven — only one role runs at any moment.
4.2 FastLeaderElection: How Votes Are Compared
The entire "data freshest wins" logic in ten lines:
// Compare zxid first, then myid
protected boolean totalOrderPredicate(
long newId, long newZxid,
long curId, long curZxid) {
if (newZxid != curZxid) {
return newZxid > curZxid;
}
// Data equally fresh, break tie with id
return newId > curId;
}zxid is the primary key; myid only breaks ties — so "give higher myid more weight" is wrong; myid size doesn't imply data freshness. Conversely, a node with lagging zxid cannot win even with max myid; its only path is sync from new Leader.
4.3 Leader.propose: How Proposals Enter Queue
Leader on receiving write request does three things: generate zxid, wrap as proposal, place in "uncommitted map" and hand to send queue . The uncommitted map ( outstandingProposals) is the basis for COMMIT decision: when enough ACKs arrive, remove from here and commit.
// zookeeper/server/quorum/Leader.java
public Proposal propose(Request request) {
byte[] data = SerializeUtils.serialize(request);
Proposal p = new Proposal();
// zxid starting point fixed after Leader election
p.packet = new QuorumPacket(PROPOSAL, request.zxid, data, null);
p.request = request;
// Uncommitted map: wait for quorum ACKs
outstandingProposals.put(request.zxid, p);
// FIFO send queue — order must not be scrambled
sendAckRequest(p);
return p;
}Why FIFO mandatory? ZAB guarantees "sequential consistency", not "eventual consistency in any order". If proposals arrive out of order, Followers apply transactions in wrong order — e.g., delete parent then create child → DataTree breaks. In all state-machine replication protocols, FIFO channel is a correctness prerequisite, not a performance optimization.
4.4 Follower.processPacket: Persist and ACK Sequence
// zookeeper/server/quorum/Follower.java
case Leader.PROPOSAL:
// 1. Write transaction log (fsync before considered persisted)
zk.getZKDatabase().append(qp, true);
// 2. Pre-apply (rollbackable)
zk.getZKDatabase().processTxn(qp, true);
// 3. Send ACK (the real promise)
ack.setZxid(qp.getZxid());
writePacket(ack, true);
break;Order cannot change: persist first, pre-apply second, ACK last . If ACK before persist, Leader may gather quorum and reply "success" to client, then this machine loses power — log not written, memory not changed, data permanently lost. "Pre-apply" means write to in-memory DataTree but mark uncommitted ; on subsequent COMMIT mark committed, on TRUNC roll back. This is the implementation point of recovery guarantee 2 "uncommitted never applied".
4.5 Snapshots and Logs: State Compression
txnlog = incremental, snapshot = full; together they bound restart time:
Every snapCount transactions (default 100k) trigger async snapshot .
After snapshot completes, older txnlogs can be deleted.
Snapshot filename embeds last zxid; recovery: "load max-zxid snapshot, replay logs with higher zxid" — single-step positioning.
Two production pitfalls:
Snapshot is async, done by separate thread (fork), briefly consumes large memory — small heap may trigger GC, threatening sessions (recalls incident 2).
txnlog must reside on dedicated disk (dataLogDir) , separate from snapshots and system logs. Log writing is pure sequential append + fsync — the latency hotspot. Contending for IO with random-read system disk multiplies write latency several times.
Advanced Topics: Boundaries, Comparisons, and Pitfalls
5.1 Why Crash Recovery's Two Guarantees Are Hard
The two guarantees pull in opposite directions:
"Committed never lost" → New Leader must have most data.
"Uncommitted never applied" → New Leader must force newer Followers to truncate.
They only coexist without contradiction under "zxid total order + majority" : any committed transaction exists on a majority; new Leader elected by majority; two majorities must intersect → new Leader has seen all committed transactions.
"Two majorities always intersect" is the most important sentence in distributed consistency — Paxos, Raft, ZAB all build on it. Conversely, if minority could commit, two disjoint sets could commit conflicting writes → not CP.
5.2 ZAB vs Raft: Similar but Different
Same: majority commit · log replication · term/epoch<br/>Diff 1: ZAB elects by data freshness; Raft by log length + term<br/>Diff 2: ZAB has explicit sync phase; Raft catches up via AppendEntries<br/>Diff 3: ZAB Leader unilaterally COMMITs; Raft uses commitIndex<br/>Essence: Both are engineered Paxos variants
Interview answer "both Paxos variants, both use majority, both use terms" is barely passing. Real difference in recovery: ZAB has an explicit synchronization phase (DIFF/SNAP/TRUNC three sync modes) ensuring all Followers strictly match new Leader before broadcast begins; Raft has no independent sync phase, relies on subsequent AppendEntries to gradually converge. ZAB's approach gives a crisp new-term boundary at the cost of slower failover.
5.3 Using ZK as Registry: Where It Hurts
Back to article 01's foreshadowing: ZK is CP; registry needs AP . Three pain points:
Election period = entire cluster unwritable — services cannot register or renew; existing sessions don't expire immediately but new instances can't join.
Partition = minority partition refuses service — if datacenter splits, smaller half's services can't write even to local ZK; must wait for network heal.
Write bottleneck = single Leader — all registrations and renewals go through Leader's fsync; at 10k+ instances, Leader fsync pressure is real.
This is why Nacos uses Distro (AP) for ephemeral instances and Raft (CP) for persistent ones (ref SC article 02). But conversely, distributed locks, leader election, strong-consistent config reads precisely need ZK's CP semantics — a lock allowing "both sides acquire" isn't a lock. Tools have no superiority, only scenario fit — truer in coordination layer than communication layer.
5.4 Session Expiry: Why GC Pause Kills Cluster
Session essence: "server keeps accounts for you". Client pings every sessionTimeout/3; server's SessionTracker maintains expiry-sorted structure, periodically scans and evicts expired sessions.
Key: Scan uses server clock; unrelated to whether session is truly "dead" . Client Full GC 8s, container CPU throttled, or thread pool saturated by business — all prevent PING; server expires regardless.
Parameter constraint often missed: sessionTimeout proposed by client but clamped by server to [2×tickTime, 20×tickTime] . Default tickTime=2000ms → min 4s, max 40s. Configuring 60s on client yields 40s actual — classic "configured but not effective" pitfall . Production advice: keep tickTime=2000, sessionTimeout=10–20s, keep server JVM GC pauses <1s (G1 + reasonable heap, avoid hundreds of GB heaps).
Another asymmetry: Follower expires sessions more conservatively than Leader — only after local timeout exceeds ~half the session timeout without Leader expiry notification does it self-clean, avoiding false global session kills from brief disconnects. Explains transient inconsistency where "ephemeral node exists on Follower but gone on Leader" is normal.
5.5 Disk, Memory, and Capacity
Three hard numbers define ZK's capacity profile:
Single node data limit 1MB (exceed → error; don't store large objects).
Full dataset must fit in memory (DataTree entirely in heap; capacity limited by heap, not disk).
Write throughput bounded by fsync (HDD ~ few thousand TPS; SSD 10–20k).
Thus ZK's positioning is clear: Store small critical metadata, not business data .
Disk : dataLogDir on dedicated SSD; disable swap ( vm.swappiness=0) — swap turns fsync into hundreds of ms; keep 3+ days disk headroom, full disk = incident 3.
Memory : Heap = data size × 2–3 headroom; avoid huge heaps — GC pauses directly threaten sessions; recommend G1 with max pause target ~200ms.
Connections : One session per client; file descriptors reserve "client count × 1.2"; at 10k+ clients watch maxClientCnxns per-IP limit (default 60; NAT'd clients hit this).
Counter-intuitive ops advice : Don't let ZK cluster "never change term" . Long-running clusters accumulate counter, txnlogs grow, snapshots enlarge, recovery silently degrades from seconds to minutes. Periodic rolling restarts (Followers first, Leader last) both validate recovery path and reset counter — turn "failure drill" into routine maintenance, better than discovering 3-minute recovery during real outage .
5.6 High-Frequency Pitfall Checklist
Using ZK as high-frequency config center : Every read creates connection, every write triggers fsync → Leader saturated. Config should be "read once + Watch for changes", not polling.
Creating children under ephemeral node : Direct error. Design requires instance nodes to be leaves.
Duplicate myid : Two machines same myid → inflated vote counts, log corruption. Deployment scripts must enforce uniqueness.
Cross-datacenter even deployment : Three DCs, one node each → any DC network blip loses quorum. Correct: "2+2+1" or dual-DC "3+2" ensuring either side retains majority.
Ignoring maxClientCnxns : Containerized pods share egress IP; 60-connection limit triggers storm of errors.
Storing business data under root : Thousands of znodes flat under root → management chaos, Watch granularity lost. Use /app/env/type/id hierarchy.
Assuming reads are always latest : Follower local read may lag; strong consistency requires sync() or read from Leader.
Sending large payloads without tuning jute.maxbuffer : Default 1MB limit; exceed → connection dropped with cryptic error.
Learning Path and Interview Framework
6.1 Four-Stage Roadmap
Beginner (3 days) : Local 3-node pseudo-cluster (three dataDir, myid, port sets); use zkCli to create four node types; kill Leader, observe election logs and state transitions.
Intermediate (1 week) : Use four-letter commands (stat/mntr/cons/srvr) to monitor request rates and connections; deliberately fill txnlog disk to reproduce incident 3, watch quorum exit.
Deep (2 weeks) : Read QuorumPeer.run, FastLeaderElection, Leader.propose, Follower.processPacket source code; draw "proposal-ACK-COMMIT" three-queue flow diagram.
Practice (ongoing) : Attach ZK registry (Curator) to article 01's mini-RPC; inject 200ms latency and 5% packet loss via tc netem; observe election and reconnect behavior; write your own postmortem.
6.2 Answer Framework: ZK Consistency in Five Acts
Asked "How does ZK guarantee consistency?", don't start from znode. Use five acts:
Positioning — CP state-machine replication; core: all nodes apply same writes in same order.
Roles & States — Leader/Follower/Observer; LOOKING/FOLLOWING/LEADING.
Write Path — Via Leader for zxid ordering, FIFO broadcast, disk ACK, majority COMMIT.
Recovery — Elect max-zxid Leader; DIFF/SNAP/TRUNC alignment; two guarantees.
Cost — Election downtime, write bottleneck, read staleness. Five acts deliver depth naturally.
6.3 Twenty Interview Questions (with Answers)
The article provides detailed Q&A covering: ZK definition, four node types, ZAB two phases, zxid structure, Leader-only writes, read staleness, election criteria, majority anti-split-brain, 3 vs 5 node choice, Observer role, session expiry mechanics, sessionTimeout clamping, Full GC cascade, txnlog/snapshot relationship, fsync-before-ACK rationale, recovery discovery/sync details, ZAB vs Raft core differences, Watch pitfalls, registry defects, and production monitoring metrics (five categories: role/leader presence, connections/sessions, request latency especially write P99, disk/fsync time, txnlog space and snapshot duration; alert on "node in LOOKING >30s" half-dead state).
6.4 One-Page Cheat Sheet
Three states: LOOKING / FOLLOWING / LEADING<br/>Two phases: Crash Recovery / Message Broadcast<br/>Write path: Via Leader → zxid → Majority ACK → COMMIT<br/>zxid = epoch(high32) + counter(low32)<br/>Election: Compare zxid first, then myid<br/>Quorum = N/2+1 · 5 nodes tolerate 2<br/>Two iron laws: Committed never lost, Uncommitted never applied<br/>Process alive ≠ Cluster available
Summary
Compress this article into three sentences:
Everything in ZooKeeper revolves around "assigning a global order to write requests" — Leader, zxid, FIFO, majority ACK all serve this.
Availability and consistency share one switch — quorum — it prevents split-brain but also means fewer nodes directly reduces write capacity.
Recovery's two guarantees are the protocol's value — committed never lost, uncommitted never applied — achieved by zxid total order plus intersecting majorities.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
