Skip to content
Distributed Locks & Leader Election with Curator: Pitfalls, Fair Locks & Fencing Tokens

Distributed Locks & Leader Election with Curator: Pitfalls, Fair Locks & Fencing Tokens

This article analyzes distributed locks and leader election using Apache Curator, detailing three failure scenarios (false release, herd effect, lock loss), fair vs herd lock implementations, fencing tokens for correctness, Curator's lock family, source code walkthroughs, and practical pitfalls with interview Q&A.

Architecture Pony
Architecture Pony
Architecture Pony

Three Failure Scenarios: How Distributed Locks Break

The article opens with three real-world accidents that illustrate fundamental problems with ZooKeeper-based locks:

False Release (Lock Timeout While Task Runs) : A scheduled task holds a ZK lock with a 3-second acquire timeout. The task runs for 40 seconds, but its session expires after 15 seconds (due to GC pause), causing the server to delete the lock node. Another task acquires the lock and both process the same refund batch, causing duplicate refunds. Lesson: Lock validity is bound to session lifetime, not task execution time.

Herd Effect (Thundering Herd on Unlock) : 200 workers watch the same lock node. When released, all 200 receive notifications and attempt to create the lock simultaneously, spiking CPU and increasing latency from 5ms to 200ms. This is a herd lock — one unlock notifies everyone. Switching to a fair lock (watching only the immediate predecessor) reduces notifications to one.

Lock Loss Without Client Awareness : During a 20-second network blip, a session expires and the lock is transferred to another node. The original client's lock object remains locally, so it continues writing to the ledger, causing data corruption. Root cause: Lock ownership lives on the server; the client object is just a cached view that can stale.

Three corresponding lessons: (1) Lock validity follows session timeout — business execution must fit within it or renew; (2) Unlock notification scope determines scalability — fair lock is the engineering default; (3) Any distributed lock can fail — critical writes must carry idempotency or fencing tokens.

Technical Analysis: From Mutual Exclusion to Leader Election

2.1 Three Implementation Patterns, Differing in "Who Goes First"

Pattern 1: Exclusive Ephemeral Node (Herd Lock) — Everyone competes for the same path; create success = lock held; others watch that node. Simple but triggers herd effect on release.

Pattern 2: Ephemeral Sequential Nodes Queue (Fair Lock) — Each client creates an ephemeral sequential node under /lock. Smallest sequence number holds the lock; others watch only their immediate predecessor. One unlock wakes exactly one waiter. This is Curator's InterProcessMutex.

Pattern 3: Semaphore / Read-Write Lock — Relax "smallest holds" to "first N hold" for semaphores (rate limiting); split N into "write=1, read=unlimited" for read-write locks. All share the same mechanism; only maxLeases differs.

Curator's InterProcessSemaphoreMutex (non-reentrant mutex) is InterProcessSemaphoreV2 with maxLeases=1; InterProcessReadWriteLock uses two internal mutexes sharing a parent path: write lock maxLeases=1, read lock maxLeases=Integer.MAX_VALUE.

2.2 Why Ephemeral Nodes Are Mandatory

Persistent nodes survive kill -9, leaving the lock forever stuck — cleaning them requires another lock, causing recursive deadlock. Ephemeral nodes bind release to session lifecycle, enabling automatic cleanup. The trade-off: session survival depends on heartbeats, which GC pauses, CPU throttling, or network jitter can break.

Engineering alignment required: sessionTimeout ≥ max business execution time OR renew during lock hold . ZK renews automatically via client PINGs (simpler than Redis manual renewal), but if the process stalls and cannot send PINGs, renewal stops. Real criterion: Can your business go a full sessionTimeout without any ZK interaction? If yes, shorten lock duration or split tasks.

2.3 Herd Lock vs Fair Lock: How Many Notified on Unlock?

Difference in one sentence: Herd lock watches the lock node itself; fair lock watches its own predecessor. With N contenders, herd lock generates N notifications + N create attempts (N-1 guaranteed failures); fair lock generates 1 notification + 1 check. Fair lock's acquisition check is a read (local comparison), not a write — herd lock's N-1 failures are write requests that must go through Leader and fsync (per article 07), amplifying the gap under write pressure.

2.4 Three Lock Semantics Issues

Reentrancy : InterProcessMutex is reentrant (ThreadLocal counter); InterProcessSemaphoreMutex is not. Often overlooked until nested calls deadlock.

Timeout : acquire(timeout) returns false on timeout (caller must handle); no-arg acquire() waits indefinitely — if predecessor stalls but session lives, you wait until its expiry. Infinite wait is almost always wrong in production.

Lock Loss : On LOST state, Curator revokes local lock state, but your business code keeps running . Must self-check periodically (e.g., every 100 records, checkExists own node) and abort/rollback if gone.

Curator guarantees "lock object state matches server"; it does not guarantee "business stops when lock expires" — that's application responsibility. Accident 3 conflated the two.

2.5 Leader Election Trio: Latch, Selector, Manual

LeaderLatch : All create ephemeral sequential nodes; smallest becomes leader; others watch predecessor. Provides hasLeadership() and await(). Leader holds until it crashes — suits "long-lived leader" scenarios.

LeaderSelector : Same smallest-node election, but leadership logic lives in takeLeadership() callback. Returning from callback releases leadership ; with autoRequeue() it re-queues. Suits "rotating work" like sharded task scheduling.

Manual : Just fair lock's smallest-node check without release. Interview "write ZK leader election" expects this fair-lock algorithm.

All three share: ephemeral sequential node + smallest sequence + watch predecessor . Leader election and fair lock are the same algorithm; difference is only whether you release after work. ZK suits leader election because its lock mechanism already provides total ordering.

2.6 Fencing Token: Last Line of Defense When Locks Fail

Hard fact: A lock holder may keep writing while the lock has already moved to another (accidents 1 & 3). This is not a ZK flaw — any lease-based lock (Redis, etcd, DB row locks) has this window .

Solution: Attach a monotonically increasing token to every write ; storage rejects writes with tokens ≤ previously seen. ZK provides this natively: zxid (global monotonic, per article 07) or the sequential node number (monotonic under same parent). This is the fencing token .

Write Pattern 1 (Recommended) : On lock acquisition, write the sequential node number as token into the resource row ( UPDATE ... SET owner_token=? WHERE id=? AND owner_token<?). Old holder's writes rejected by DB.

Write Pattern 2 (Lightweight) : Store owner ID + verify node still exists before write. Has a race window but better than nothing.

Fencing token downgrades correctness from "lock service never fails" (impossible: GC, partitions, clocks) to "storage does one comparison" (a single SQL condition). Every "write money, inventory, state machine" scenario needs this layer.

2.7 Curator Lock Family Quick Reference

InterProcessMutex     Mutex · Reentrant · Most used
SemaphoreMutex        Mutex · Non-reentrant
InterProcessSemaphoreV2 Semaphore · maxLeases=N
ReadWriteLock         Write 1 / Read MAX
MultiLock             Multiple locks held simultaneously
LeaderLatch           Leader election · Hold forever
LeaderSelector        Leader election · Can yield

Selection mnemonic: Mutex for mutual exclusion (default reentrant), Semaphore for rate limiting, ReadWriteLock for read/write separation, LeaderLatch for fixed leader, LeaderSelector for rotating leader. MultiLock only for "hold multiple resources simultaneously"; it sorts paths internally to avoid most ordering deadlocks, but cross-system combos still need consistent ordering.

Core Flow: One Lock Acquisition Journey

Five contenders (A–E) for /lock using InterProcessMutex:

T+0  A/B/C/D/E each create ephemeral sequential nodes
     /lock/_c_0000000001 ~ 05
T+1  Each getChildren and sort
T+2  A sees it is 01 → acquires lock
T+2  B–E watch only their immediate predecessor
     B→01  C→02  D→03  E→04
T+3  A works… (others silent)
T+9  A releases: deletes 01
T+9  Only B notified (single wake)
T+9  B re-checks list: now smallest → acquires
T+9  C need not change watch target (its watched 02 still exists)
T+15 B releases → C wakes… chain continues

Key Point 1 : Every wake triggers a full list re-check — cannot assume "predecessor gone = I'm next" because others may have abandoned or expired. Re-check asks "am I smallest?" not "am I first arrived?" — sequential node names encode order.

Key Point 2 : Watch target always points to "current predecessor". If predecessor disappears (it crashed, and its predecessor also crashed), Curator re-selects predecessor and re-registers watch in next loop — "watches can expire" remedied by "wake up and recompute" .

Key Point 3 : Release = delete own node, nothing else. No explicit "notify next" call — server-side deletion triggers watch, next wakes naturally. This "data-change-driven" design eliminates extra coordination channels, making ZK locks more reliable than custom notification.

Key Point 4 : Must distinguish why a node disappeared. Self-delete (normal release) vs predecessor gone (my turn) both change the list, but former means "I shouldn't hold lock", latter means "I should acquire". Curator uses ourPath existence: if my own node is gone, lock is lost — throw exception, don't continue . This is where accident 3 gets caught at framework level.

Core Source Code: Six Segments to Understand Fair Lock & Leader Election

4.1 Exclusive Lock: Ten-Line Anti-Pattern

// Naive implementation: also a herd lock
try {
  zk.create("/lock", null, OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
  return true;
} catch (NodeExistsException e) {
  zk.exists("/lock", true); // watch this node
  return false;
}

Works and provides mutual exclusion, but three fatal flaws: non-reentrant (same thread second call fails), herd lock (all watch same node, unlock wakes all), unfair (whoever grabs it wins, starvation possible). Only fit for "low contention, ultra-short hold" like one-time init. For production, see next.

4.2 Fair Lock Manual: Ephemeral Sequential + Watch Only Predecessor

// 1. Create ephemeral sequential node (get queue number)
String me = zk.create("/lock/lock-", null, OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL);
while (true) {
  // 2. On every wake, re-fetch and sort
  List<String> kids = zk.getChildren("/lock", false);
  Collections.sort(kids);
  int i = kids.indexOf(nameOf(me));
  if (i == 0) {
    return; // smallest: got lock
  }
  // 3. Watch only predecessor, avoid herd
  Stat s = zk.exists("/lock/" + kids.get(i - 1), true);
  if (s == null) {
    continue; // predecessor gone: recompute
  }
  awaitNotify(); // wait for delete event
}

These 30 lines are the entire ZK fair lock skeleton: create node → sort → if smallest hold → else watch predecessor → wake and recompute . Curator's InterProcessMutex productionizes it: reentrancy counting, timeout control, connection state handling, exception retry, and "own node gone = lock lost" judgment. Understanding these 30 lines maps every framework behavior.

4.3 Curator: attemptLock Three Steps

// LockInternals.attemptLock (simplified)
// 1. Create ephemeral sequential node
ourPath = driver.createsTheLock(client, path, nodeData);
// 2. Loop to check if smallest
boolean hasLock = internalLockLoop(start, timeout, unit);
// 3. Timeout failed: delete own candidate node
if (!hasLock) {
  client.delete().guaranteed().forPath(ourPath);
}
return hasLock;

Step 3 is critical but often missed: On timeout failure, must delete own candidate node , otherwise it occupies a queue slot forever; subsequent waiters stall until its session expires — observed as "lock mysteriously stuck for seconds then recovers". This is "quit queue but clean up" etiquette, missing in many manual implementations.

4.4 internalLockLoop : Wake-Up Decision Logic

while (!haveTheLock) {
  List<String> children = getSortedChildren();
  String seq = extractSeq(ourPath);
  int index = children.indexOf(seq);
  if (index < 0) {
    // Own node gone: lock lost
    throw new Exception("node gone");
  } else if (index == 0) {
    haveTheLock = true;
  } else {
    // Watch predecessor: delete wakes
    String prev = children.get(index - 1);
    client.checkExists().usingWatcher(w).forPath(path + "/" + prev);
  }
}

Three branches map exactly to three states: index < 0 = lock lost (throw, catches accident 3); index == 0 = I'm smallest; else = wait predecessor . No sleep, no polling — purely watch-event-driven, wake and recompute. "Event-driven + wake-recompute" is the standard paradigm for distributed coordination , saving resources vs timed polling.

4.5 getsTheLock : How maxLeases Becomes Semaphore

// StandardLockInternalsDriver
public PredicateResults getsTheLock(CuratorFramework c, String path, int maxLeases, String node) {
  List<String> children = getSortedChildren(c, path);
  int idx = children.indexOf(node);
  // Key line: first N get the lock
  boolean gets = idx < maxLeases;
  String toWatch = gets ? null : children.get(idx - maxLeases);
  return new PredicateResults(toWatch, gets);
}

The line idx < maxLeases is the switch for the entire lock family: maxLeases=1 → mutex; maxLeases=N → semaphore (limit N concurrent); read lock gets MAX_VALUE → "read-read non-exclusive" . toWatch isn't "previous one" but "N-th previous" — with capacity 3, the 4th waiter watches the 1st. One parameter, one judgment, powers Mutex, Semaphore, ReadWriteLock — Curator's most elegant design.

4.6 LeaderLatch: Leader Election = Lock That Never Releases

// LeaderLatch.start internals: create sequential node
ourPath = client.create()
  .creatingParentContainersIfNeeded()
  .withProtection()
  .withMode(EPHEMERAL_SEQUENTIAL)
  .forPath(latchPath + "/_c_");
// Then same as fair lock: smallest wins
// Difference: after winning, DO NOT RELEASE, hold forever
// Until process exits / session expires
// Business side
latch.await(); // block until elected
if (latch.hasLeadership()) {
  doWorkAsLeader();
}

LeaderLatch and fair lock code are nearly identical — because they are the same algorithm . Only difference: fair lock deletes own node after work (yields); LeaderLatch never deletes (holds). So "can you write leader election" reduces to "can you write fair lock", and "leader election correctness" reduces to "will leader's node disappear when it crashes" — exactly what ephemeral nodes guarantee.

Advanced Topics: Correctness, Comparison & Pitfalls

5.1 Leader Election's "Dual-Leader Moment" — Why Unavoidable

Between old leader crash and new leader election, there's a window: old leader merely "thinks it's still leader" (network partition, GC pause) but isn't actually dead. ZK guarantees new leader election uniqueness (smallest node uniqueness), but cannot force old leader to stop immediately — it may be disconnected from ZK and unaware it lost leadership.

Correct pattern is dual insurance : (1) Leader logic must be idempotent/replayable (e.g., only "assign tasks"); (2) Before each critical action, verify still leader ( hasLeadership() or checkExists own node) and use fencing token so downstream rejects old leader. "Make old leader vanish" is impossible; "make old leader's writes invalid" is achievable — two engineering forms of the same principle.

5.2 ZK Lock vs Redis Lock: Not a Performance Contest

ZK Lock   CP · Ephemeral nodes · Fair queue
          10-20k TPS · Millisecond latency
Redis Lock AP · SETNX + TTL
          100k+ TPS · Sub-millisecond
Trade-off: Correctness first → ZK
           Throughput first → Redis
           Both need fencing as safety net

Standard interview answer: Explain semantic differences first, then scenarios . ZK lock holds for session duration — won't auto-release due to "business timeout" (only session expiry), and won't lose lock due to async replication. Redis master failover can lose locks (RedLock controversy), but latency is an order of magnitude lower. Money-writing scenarios → ZK or DB; high-concurrency low-value (coupons, flash sales) → Redis .

Addendum: ZK write throughput ~10-20k TPS (fsync cost, per article 07), so unsuitable for "high-frequency short locks" — tens of thousands of lock/unlock per second saturates Leader. Hence many teams: "ZK for leader election (low freq), Redis for mutex (high freq)" — coexistence is fine.

5.3 Lock Capacity & Scale

Three numbers to remember: (1) Each waiter consumes one ephemeral node — 1,000 threads = 1,000 nodes + 1,000 watches, fine; but "one lock per order" (100k orders) = 100k nodes, bloats DataTree and WatchManager — use sharded locks or DB row locks instead. (2) Lock nodes are writes — acquire + release = 2 writes, both go through Leader + fsync. (3) Retry must be bounded — infinite acquire() during holder stall causes "whole chain waits for him" avalanche.

Engineering rule: Lock QPS × 2 (acquire + release) should be far below 1/10 of cluster write capacity . Beyond that, switch implementation (Redis/DB) or redesign (reduce granularity, lock-free CAS, sharding). "Using locks to solve concurrency" is often the most expensive solution — first ask if a lock-free design exists.

5.4 High-Frequency Pitfall Checklist

Business time > sessionTimeout : Lock expires before work finishes. Shorten tasks or batch with periodic lock verification.

Using no-arg acquire() (infinite wait) : Predecessor stalls but session lives → wait until expiry (tens of seconds). Production must use timeout and handle false return.

finally release without null-check : acquire fails → release throws IllegalMonitorStateException, swallowing original exception — debugging shows only "mysterious lock exception".

Passing lock object across threads : InterProcessMutex reentrancy is thread-local; thread A locks, thread B releases → error or worse, fails to release.

Assuming LOST auto-stops business : It doesn't. Must self-verify periodically or before critical writes.

Writing money without fencing token : Ultimate form of accidents 1 & 3. Storage-side "token monotonic" condition blocks most duplicate writes.

LeaderSelector callback never returns : Main logic loops forever inside callback → "yield" never happens, autoRequeue breaks. Callback should "do chunk of work and return".

Locking read-modify-write but reading stale data : Follower local read may lag (per article 07). First read after lock may hit stale follower, compute on old value, write back. Lock scenarios should read Leader or call sync first.

5.5 Full Postmortem: One Duplicate Refund Incident

Timeline of accident 1 expanded:

T+0 : Node A acquires lock, starts refund batch (5,000 orders, est. 40s).

T+14 : At 1,800 orders, A hits 6s Full GC; cumulative GC pauses exceed 15s sessionTimeout.

T+15 : Server declares A session expired, deletes lock node.

T+15 : Node B's watch fires, re-checks list, finds itself smallest, acquires lock, starts same batch.

T+20 : A's GC ends, continues from order 1,800.

Now A and B both running — A's lock object still shows "held" because its ZK client may have reconnected with new session while local mutex remained unseen by business.

Root causes in three layers, fixes in three layers:

Business layer — Batch → sharded + record progress per shard, idempotent re-run (dedupe by order ID) so dual-run doesn't double-refund.

Framework layer — During hold, every 100 orders checkExists own node; if gone, abort immediately.

Storage layer — Refund ledger adds unique index (order_id + batch_id) + write fencing token; old leader's writes rejected by DB.

Cost rises, reliability rises across layers. Engineering practice: usually do all three, but at minimum do layer 2 — it's the only point that can halt business the instant lock is lost.

Learning Path & Interview Prep

6.1 Four-Stage Roadmap

Beginner (2 days) : Write raw API exclusive lock and fair lock; two processes alternate, print wake timestamps; see "herd" vs "single wake" difference firsthand.

Intermediate (1 week) : Use Curator's Mutex / Semaphore / ReadWriteLock each in a demo (limit 3 concurrent, read-write split); kill -9 to verify ephemeral auto-release.

Deep (2 weeks) : Read

LockInternals
attemptLock

/ internalLockLoop / getsTheLock; draw maxLeases 1→N evolution diagram; read

LeaderLatch
checkLeadership

loop.

Practice (ongoing) : Add fencing token to any "distributed lock + DB write" in your project: write sequential node number into resource row, update with token > owner_token condition; verify old leader writes rejected with two machines alternating lock.

6.2 Interview Answer Framework: Distributed Lock Five-Part Structure

Asked "How to implement distributed lock with ZK?" answer in five parts:

Primitives — Ephemeral sequential nodes (auto-release + natural queue).

Algorithm — Create node, sort, smallest holds, others watch predecessor.

Semantics — Reentrancy via local counter; timeout via acquire(timeout); lock loss via "own node gone" detection.

Correctness Boundary — Session expiry causes false release; need fencing token so downstream rejects old holder.

Trade-offs — CP guarantees correctness but write throughput limited; high-frequency short locks better on Redis; leader election must handle dual-leader moment with idempotency.

Part 4 (fencing) shows depth — interviewers often wait for this.

6.3 Twenty Interview Questions (with Answers)

The article provides 20 Q&A pairs covering core concepts: ZK lock core idea (Q1), why ephemeral nodes (Q2), herd effect & fair lock avoidance (Q3), fair lock steps (Q4), why re-check list on wake (Q5), false release conditions (Q6), fencing token definition (Q7), InterProcessMutex reentrancy (Q8), semaphore vs mutex implementation diff (Q9), LeaderLatch vs LeaderSelector (Q10), dual-leader possibility & handling (Q11), why delete candidate node on timeout (Q12), why avoid infinite acquire() (Q13), how business detects lock loss (Q14), ZK vs Redis selection (Q15), 100k locks scalability (Q16), MultiLock deadlock risk (Q17), ZK read-write lock implementation (Q18), why watch read staleness in lock (Q19), production validation methods (Q20).

6.4 One-Page Cheat Sheet

Lock primitives: Ephemeral + Sequential + Watch predecessor
Acquire: Create node → Sort → Smallest wins
Release: Delete own node (no notify call)
Timeout fail: MUST delete candidate node
Lock lost: Own node gone → Throw error
Fencing: Token monotonic, old holder writes rejected
Leader election = Fair lock that never releases
One iron law: Lock is a lease; correctness lives in token

Summary

Three closing statements:

ZK locks have only one algorithm — Ephemeral sequential queue + smallest holds + watch predecessor; mutex, semaphore, read-write lock differ only in maxLeases parameter.

Lock is a lease, not ownership — Session expiry = lock loss; business duration must align with session timeout; long tasks must self-check.

Correctness's last mile is in storage — Fencing token turns "old holder's write" into a detectable, rejectable single comparison — the mandatory safety net for all distributed locks.

With this, the Netty (6 articles) + ZooKeeper (3 articles) foundation is complete: a communication layer handling 10k+ connections, a coordination layer providing total write ordering, and lock/leader-election primitives with auto-cleanup. The finale "Build Your Own RPC Framework — Netty Transport + ZK Registry + Custom Wheel" assembles them: custom protocol, serialization, connection pool, load balancing, service discovery, heartbeat, graceful shutdown — one end-to-end runnable, benchmarkable, resume-worthy mini RPC.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

ZooKeeperInterview Preparationleader electionDistributed LocksFencing TokensHerd EffectApache CuratorFair Locks
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.