Skip to content
ZooKeeper Watch Mechanism & Curator Best Practices: Solving Herd Effect & Connection Pitfalls

ZooKeeper Watch Mechanism & Curator Best Practices: Solving Herd Effect & Connection Pitfalls

This article analyzes ZooKeeper's watch mechanism through three production accidents — one-time triggers, herd effect, and missed events during disconnection — then explains Curator's cache abstractions, session vs connection lifecycles, reconnection rebuild process, and three-layer solutions to herd effect with troubleshooting commands and interview questions.

Architecture Pony
Architecture Pony
Architecture Pony

Three Accidents, Three Lessons

The previous article dissected ZooKeeper internals: quorum, zxid, two-phase broadcast. But in engineering you rarely touch those directly — you touch the client. And the client layer has denser pitfalls: watches are one-time, notifications carry no data, a broken connection doesn't necessarily mean a dead session, and a dead session wipes all ephemeral nodes. The previous article's accident #2 (GC pause causing mass disconnect) has a mirror version on the client side: wrong watch registration timing causes all changes during disconnection to be missed entirely.

This article follows "how to be a correct client": native API vs Curator trade-offs, the full chain of a single watch notification, four watch properties, two lifecycles of session and connection, the complete disconnection/reconnection process, correct usage of Curator's three caches, and the watch system's most famous scale problem — the herd effect.

Three accidents map to three lessons: ① Watch fires once — register once, get notified once; ② Wrong watch granularity causes herd effect at scale; ③ Events during disconnection are lost — after reconnect you must do a full reconciliation, not wait for notifications. These are the three main threads of this article.

I. Accident Scenes: Three Ways to Die

Accident 1: Config Only Hot-Reloaded Once

Implemented config hot-reload with native API: register watch → receive NodeDataChanged → read new value. After deployment, the first change took effect; from the second onward, never again. Root cause: ZooKeeper's watch is one-time trigger — the moment the server sends the notification, it removes that watch from its table; the callback only read data and didn't re-register, so the path was broken forever. This is not a bug, it's by design , but misuse turns it into an accident.

Accident 2: One Change Takes Down the Whole Cluster

At 1,200 instances, every release caused ZK cluster network traffic to spike for seconds, with occasional client timeouts. Root cause: herd effect — 1,200 clients all watched the same parent node; one change triggered 1,200 notifications and 1,200 getData read-backs, instantly saturating server egress bandwidth and session threads. The problem isn't volume, it's watch granularity — everyone watching the same door, and when it opens everyone rushes in at once.

Accident 3: Disconnected for Five Seconds, List Wrong for an Hour

Client network jitter caused a 5-second disconnect. During those 5 seconds, an instance went down (node deleted) and a new node came up (node created). After reconnect, the consumer's local instance list remained unchanged — the deletion wasn't received, the addition wasn't received; a non-existent instance stayed in the list for an hour, traffic kept hitting a zombie node. Root cause: all change events during disconnection are lost, yet the client treated "reconnect succeeded" as "data is still fresh". After reconnect you must do a full pull reconciliation; you cannot wait for notifications.

Three accidents, three lessons: ① One-time is watch's destiny; re-registration is the basic action of using it. ② Herd effect is a granularity problem; the fix is changing "watch the front door" to "watch the window". ③ Notifications guarantee "timely", not "no loss"; if lost, active reconciliation is the safety net. Below we start from the watch implementation mechanism.

II. Technical Analysis: Complete Mechanism of Watch and Client

2.1 Three Generations of Clients: Native, zkClient, Curator

ZooKeeper clients have three generations: Native ZooKeeper Client (official JAR, crude API: manual watch re-registration, no connection state management, no retry semantics); zkClient (encapsulated watch re-registration and serialization, but community largely unmaintained); Curator (Netflix-originated, now the de facto standard — connection state machine, automatic retry, three caches, ready-made recipes for distributed locks and leader election). Conclusion is simple: new projects go straight to Curator; native API only for understanding the bottom layer.

But to explain the watch mechanism clearly, we must start from the native API — because Curator's caches essentially solidify three things: "re-registration + full reconciliation + event callbacks". Understand native, and Curator's behavior becomes fully predictable; skip native and go straight to Curator, and when problems arise you can only guess blindly.

2.2 Full Chain of a Single Watch Notification

From "client registers" to "callback executes" there are six steps:

Client calls getData(path, watcher) , request carries watcher object; registration request and data read request are in the same RPC.

Server records watcher in WatchManager's tables (path → watcher set, watcher → path set, two tables cross-referenced).

Another client modifies this node , write request goes through the ZAB process described in the previous article and commits.

FinalRequestProcessor applies the transaction, reverse-looks up the tables, triggers the watch , constructs a WatchedEvent (type + state + path — note: no data ).

Notification is pushed back to the client along the connection , and simultaneously that watch is removed from the table (one-time).

Client EventThread takes the event from the queue and executes your callback , where you typically re-getData and register again. One frequently asked point: events may be "merged" — after disconnect/reconnect, Curator receives a single rebuild trigger, not every change during disconnection, so business logic must not assume "exactly one event per change".

Two properties are directly readable from the chain:

Lightweight notifications — events carry only type and path, no data; server bears no payload transport cost (one reason ZK can support massive watches).

Ordering guarantee — client guarantees "see watch event first, then see new data", so getData in the callback always fetches the post-change value; you never see "notification arrived but value is still old".

2.3 Four Watch Properties — Not a Single One Can Be Forgotten

Property 1: One-time trigger. Notification sent = deregistered. Pre-3.6 solution was "re-register in callback"; 3.6 added addWatch (persistent watch) allowing resident watches — upgrade if you can, eliminates a whole class of pitfalls .

Property 2: Notify only, no data. After receiving event you must getData yourself. So the actual traffic of a "notification storm" is "notification + read-back" doubled, making herd effect worse than imagined.

Property 3: Client-side ordering. Events for the same client are callbacks in receive order, and the callback precedes seeing the new data — Curator's caches rely on this guarantee for internal state consistency.

Property 4: Disconnect = invalidation. Changes during disconnection are not re-sent; after session expiry, all watches for that session become invalid. This is the root of accident #3, and why Curator forces a rebuild on reconnect.

2.4 Session and Connection: Two Lifecycles

This is the pair of concepts most easily confused on the client side: Connection is TCP-layer; session is server-side bookkeeping . TCP breaks, session stays alive (server waits for you to return within sessionTimeout); session expires, ephemeral nodes deleted, watches all cleared, returned sessionId invalidated — even if TCP reconnects immediately, the server will tell you Expired, and you must rebuild everything as a new session.

In native API this maps to two state categories: Disconnected / SyncConnected (connection-layer loss and recovery, session alive) and Expired (session dead, start over). Business code must handle these two categories completely differently : on Disconnected, pause writes and wait for reconnect; on Expired, recreate ephemeral nodes, re-register watches, re-contend for locks — the "zombie instance hanging for an hour" in accident #3 is essentially treating Disconnected as Expired or vice versa.

2.5 Curator's Three Caches

Curator encapsulates "re-registration + reconciliation + callbacks" into three caches, choose by need:

NodeCache: watches a single node's data, callbacks on change. Standard solution for config hot-reload , naturally avoids accident #1 (internal auto re-registration).

PathChildrenCache: watches direct children of a directory for add/remove/update, caches child node data. Standard solution for service discovery — instance up/down is child node add/remove.

TreeCache: watches entire subtree, depth configurable. Suitable for hierarchical config , but cost = watch count = node count; large trees put heavy pressure on server.

All three caches do two critical things: full rebuild on reconnect (solves accident #3), and internal threaded callbacks (your callback runs on Curator's event thread — don't do heavy work there, or you'll block subsequent events). Since Curator 5, the three are unified into CuratorCache with consistent behavior and simpler API; old projects seeing PathChildrenCache need not panic, new projects just use CuratorCache.

One often-overlooked dimension in selection: cache initialization semantics . After start(), the first thing is a full pull (Type INITIALIZED event), so your callback must handle "batch infusion at startup" — if every CHILD_ADDED triggers a remote call, startup becomes a self-inflicted herd. Correct posture: before INITIALIZED only fill local structures; after INITIALIZED expose service externally.

2.6 Herd Effect: Granularity Problem, Not Performance Problem

Herd effect definition: one event triggers many clients to do duplicate actions. In ZK the most common form is accident #2: N clients watch the same node, one change → N notifications → N read-backs. Its destructive power scales linearly: unnoticeable at 100 nodes, network jitter at 1,000, cluster avalanche at 10,000.

The core solution idea is shrink the watch audience :

Parent watch → child watch — consumers don't watch /services/order itself, but its child nodes; when one instance changes, only clients watching that specific child get notified, the other 1,199 feel nothing.

Each client watches only the instances it actually uses — establish watches based on load-balancing assignment, not full subscription.

Combine with local cache + version comparison — periodic full pull as safety net, watch only accelerates. This third layer is the real design of most mature registries (Nacos, Dubbo): push-pull combined, push guarantees timeliness, pull guarantees correctness .

III. Core Flow: Complete Process of One Disconnect-Reconnect

Put the two lifecycles from 2.4 into a timeline (assume sessionTimeout = 15s, client disconnects at T+5, recovers at T+9):

T+5 Network break, TCP RST T+5 Callback Disconnected/SUSPENDED T+5~9 Client backs off and reconnects (1s/2s/4s) T+9 Network restored, reconnects to server T+9 Server verifies session still alive T+9 Callback SyncConnected/RECONNECTED T+9 Watch table still there: previously registered watches remain valid (this example not expired) T+10 Business side triggers rebuild for full reconciliation —— If recovery at T+20 (exceeds 15s): T+20 Server has judged session expired T+20 Ephemeral nodes all deleted, watches all cleared T+20 Client receives Expired T+20 Must rebuild session, rebuild everything

Key point 1: The first thing after reconnect is not to continue working, but to reconcile. Even if session hasn't expired, events in the disconnect window are already lost (in this example, changes between T+5~9). Curator's caches auto-rebuild on RECONNECTED, but your bare watches have no one to help you rebuild — that's why accident #3 and accident #1 often appear together.

Key point 2: Expired is the highest-priority event. Session expiry notification must be handled: at this point ephemeral nodes are gone, service registration is gone, distributed lock is lost — the sessionId in your hand is waste paper. Correct action: abandon current client object, rebuild all state with a brand-new session.

Key point 3: SUSPENDED and LOST are the two gears Curator gives you. SUSPENDED ≈ connection broken but session may be alive — pause writes to ZK, but local cache can still serve reads; LOST ≈ session likely dead — all business state involving locks and ephemeral nodes must be considered invalid. Hanging business actions on these two states is the watershed of Curator usage.

Key point 4: Reconnect storm must be prevented. When a data center network jitters, hundreds or thousands of clients disconnect simultaneously, reconnect simultaneously, rebuild simultaneously — that's the client-side herd effect. Backoff reconnect (random jitter) + staggered rebuild (local random delay of a few hundred ms before full pull) + rate-limiting reconciliation requests are standard moves, same principle as the connection storm in article 06.

IV. Core Source Code: Six Segments to Understand Both Client and Server Ends

4.1 Native Client: Standard Pattern for Registration and Re-registration

ZooKeeper zk = new ZooKeeper(host, 15000, event -> {
    switch (event.getType()) {
        case NodeDataChanged:
            // One-time: after reading must re-register
            byte[] d = zk.getData(path, true, stat);
            reload(d);
            break;
        case Expired:
            // Session dead: rebuild everything
            recreateAll();
            break;
    }
});
// First registration: true = use default watcher
zk.getData(path, true, stat);

Note the true in getData(path, true, stat): it means "register default watcher", and this default watcher in the NodeDataChanged branch re-attaches via another getData — reading data and registering watch are the same action , the semantic most easily missed in native API. Miss it, and you get accident #1.

4.2 WatchManager: Server's Two Tables

// watch.ZooKeeperServer side (simplified)
class WatchManager {
    // path -> watchers hanging on it
    HashMap<String, HashSet<Watcher>> watchTable;
    // watcher -> all paths it cares about
    HashMap<Watcher, HashSet<String>> watch2Paths;
}

Two tables are bidirectional indexes: trigger by reverse-lookup watcher via path (watchTable); on session close, reverse-lookup all paths via watcher and clean up (watch2Paths). Each "path + client connection" combination is one table entry — so 1,200 clients watching the same node means 1,200 entries, 1,200 triggers; herd effect is baked in at the data structure level.

4.3 DataTree.triggerWatch: Where One-Time Happens

WatchedEvent e = new WatchedEvent(type, state, path);
// remove is the entire implementation of "one-time"
HashSet<Watcher> list = dataWatches.remove(path);
if (list == null) {
    return;
}
for (Watcher w : list) {
    if (w instanceof ServerCnxn) {
        ServerCnxn c = (ServerCnxn) w;
        // Skip connections whose session is already closed
        if (c.isClosed()) {
            continue;
        }
        c.process(e);
    }
}

The entire implementation of "one-time" is that remove: take it out and it's gone; to receive again you must re-register. The server gives no chance to re-attach; this responsibility lies entirely on the client side — Curator's entire cache system's first mission is to automate it. Also note the isClosed filter: watchers on dead connections are not pushed notifications, a small defense against "zombie watches" writing to dead connections.

4.4 ClientCnxn.SendThread: Client's Heartbeat

The native client internally uses a dual-thread model: SendThread handles I/O and heartbeats (send packets, receive packets, PING to renew session, disconnect/reconnect), EventThread handles event callbacks (take WatchedEvent from queue and execute watcher). Your callback runs on EventThread — don't do heavy work in the callback, or all subsequent events get stuck , same iron law as Netty's "Handler must not block" (echoes article 01).

// ClientCnxn.SendThread.run (simplified)
while (state.isAlive()) {
    if (!clientCnxnSocket.isConnected()) {
        // Disconnected: back off then reconnect
        startConnect(addr);
    }
    if (idleSince() > pingInterval) {
        // Idle: send PING to renew session
        sendPing();
    }
    clientCnxnSocket.doIO(pendingQueue, watchManager);
}

This loop also explains a common phenomenon: client stuck in "submitted but unacknowledged" . Sent requests sit in pendingQueue waiting for response; connection breaks, the whole queue is invalidated, business side gets ConnectionLoss — the only caution on retry is idempotency : create may have already succeeded on server, retry would create duplicate nodes, so serious scenarios use create with sessionId semantics (Curator's Protected mode adds prefix, does checkpoint resume).

PING rhythm matches the session renewal discussed in article 07: client actively sends PING at sessionTimeout/3 , server starts suspecting after two intervals of silence. Understand this loop and "why session stays alive after disconnect" is no longer mysticism — session life or death depends solely on whether the server received any packet from you (PING counts) before timeout.

4.5 Curator: Standard Usage of Three Caches

CuratorFramework client = CuratorFrameworkFactory.builder()
    .connectString(host)
    .sessionTimeoutMs(15000)
    .connectionTimeoutMs(3000)
    .retryPolicy(new ExponentialBackoffRetry(1000, 3))
    .build();
client.start();
// Config hot-reload: NodeCache
NodeCache nc = new NodeCache(client, "/config/timeout");
nc.getListenable().addListener(() -> reload(nc.getCurrentData()));
nc.start(true);
// Service discovery: PathChildrenCache
PathChildrenCache pc = new PathChildrenCache(client, "/services/order", true);
pc.getListenable().addListener((c, e) -> {
    switch (e.getType()) {
        case CHILD_ADDED:
            addInstance(e.getData());
            break;
        case CHILD_REMOVED:
            removeInstance(e.getData());
            break;
    }
});
pc.start();

This segment is the engineering version of "registration and discovery" from article 01's RPC panorama: registration side uses create().withMode(EPHEMERAL).forPath() to create ephemeral nodes; discovery side uses PathChildrenCache to watch child node add/remove. Re-registration, disconnect rebuild, event threading — three trouble-prone things all taken over by Curator — you only have one thing left: hang business actions on the correct connection states.

4.6 ConnectionStateListener: Hang Business Actions on the Right States

client.getConnectionStateListenable().addListener((c, s) -> {
    switch (s) {
        case SUSPENDED:
            // Connection broken: pause writes
            pauseWrites();
            break;
        case RECONNECTED:
            // Session alive: reconcile
            rebuildCache();
            break;
        case LOST:
            // Session dead: ephemeral nodes gone
            // Locks invalid, registration invalid
            recreateEphemerals();
            break;
    }
});

This state-action table is the most copy-worthy code in this article: SUSPENDED only degrades, RECONNECTED only reconciles, LOST rebuilds everything . Especially the LOST branch — distributed locks, leader election, ephemeral registration nodes all must restart; services missing this branch, one network jitter produces a batch of "zombies holding non-existent locks continuing to work", which is the prerequisite for the next article's distributed lock safety.

V. Advanced Topics: Scale, Troubleshooting, and Pitfalls

5.1 Three-Layer Solution to Herd Effect

Layer 1 (mandatory): Parent watch → child watch. Don't watch /services/order, watch its children. One instance change goes from "notify all" to "notify relevant", server trigger count drops from N to 1.

Layer 2 (recommended): Subscribe on demand. Consumers only watch the instance paths they actually use, not full service subscription. Cost is maintaining subscription relationships; benefit is watch count proportional to "usage relationship" not "cluster scale".

Layer 3 (safety net): Push-pull combined. Local full cache + periodic pull reconciliation (e.g., every 30s), watch only responsible for "make changes effective faster". Push guarantees timeliness, pull guarantees correctness — Nacos and Dubbo are designed exactly this way; even if ZK jitters, worst case self-heals within one reconciliation cycle.

5.2 Troubleshooting Toolbox: From Phenomenon to Data

wchc / wchs / wchp: Four-letter commands to inspect watches: wchs for total count and summary, wchc lists watch paths by session (query "what is this client watching"), wchp lists sessions by path (query "who is watching this node"). Herd effect's smoking gun is in wchp: a single path with thousands of sessions hanging.

cons / stat: Inspect connection session timeout, lastZxid, pending state; client logs' Session established / session closed can reconstruct every disconnect timeline.

Client-side logs: Curator's ConnectionState changes should be logged at INFO level and alerted — frequency of SUSPENDED/LOST is a direct indicator of network quality; normally should trend toward zero.

Three steps to troubleshoot herd effect: ① wchp to find the most-watched path; ② evaluate that path's change frequency × watch count = notification storm volume per change; ③ change parent watch to child watch or on-demand subscription, then use wchs to observe total count drop . These three steps on clusters under 1,000 clients can be done within an hour.

5.3 Watch Server-Side Cost and Capacity Planning

A single watch table entry costs little: path string + watcher reference + two HashSet slots, few hundred bytes — 100k watches ≈ tens of MB memory, usually not a problem . The real hidden CPU costs to watch are two:

First, traversal and per-connection write on trigger — one change with M watchers means M reverse-lookups and M outbound writes.

Second, trigger happens inside the transaction application thread (FinalRequestProcessor triggers watches synchronously after applying transaction, then responds); high-frequency changes on a heavily-watched node amplify "commit slows down" across the entire write path.

So capacity planning looks at the product, not the stock: Watch count × change frequency = trigger cost per unit time . Two engineering numbers: single-path watches exceeding 500 should consider granularity refactor; single-path change frequency > 1/sec with watch count > 100 is a high-risk combo. Monitoring with wchs total curve + spike observation at release moments is far safer than gut estimates.

5.4 High-Frequency Pitfall Checklist

Pitfall 1: Heavy work in callback. EventThread is single-threaded; callback blocks = all events stuck. Offload heavy work to thread pool (echoes Netty iron law).

Pitfall 2: Native watch not re-registered. Callback must re-register after reading data, or just use NodeCache/CuratorCache.

Pitfall 3: No rebuild after reconnect. Disconnect window events are not re-sent; "reconnect succeeded" ≠ "data is latest".

Pitfall 4: LOST branch doesn't rebuild ephemeral state. Session dead → ephemeral nodes gone, locks and registration invalidated, must rebuild.

Pitfall 5: TreeCache watching large tree. Watch count = node count; ten-thousand-node subtree blows up server WatchManager.

Pitfall 6: Treating retryPolicy as cure-all. Curator retry only covers ConnectionLoss and similar retryable exceptions; SessionExpired is not covered — ephemeral state rebuild relies on ConnectionStateListener, not retry.

Pitfall 7: Multiple watcher objects sharing one znode causing semantic confusion. Native API supports separate registration/trigger per watcher, but many assume "delete one, all gone". Advice: one watcher per node, no fancy tricks.

Pitfall 8: Old client connecting to new server, persistent watch missing. 3.6's addWatch requires client upgrade too; mixed use falls back to manual re-registration, don't discover via error.

VI. Learning Path and Interview Questions

6.1 Four-Stage Path

Beginner (3 days): Write watch demo with native API, deliberately skip re-registration to reproduce accident #1; use zkCli to modify data and observe notifications, experience "one-time" and "no data".

Intermediate (1 week): Implement article 01's registration and discovery with Curator (NodeCache + PathChildrenCache); use iptables to cut network for 10s, observe SUSPENDED→RECONNECTED→rebuild log sequence.

Advanced (2 weeks): Read WatchManager, DataTree.triggerWatch, ClientCnxn source code; use wchp in load test to quantify herd effect, implement "parent→child" solution and retest.

Practice (ongoing): Do a health check on project's ZK usage: grep all registerWatcher/getData, verify against pitfalls 2, 3, 4 one by one; hook ConnectionState logs into alerting.

6.2 Answer Framework: Watch Questions in Four Parts

When asked "Explain ZK's watch mechanism", answer in four parts: ① Mechanism — client registration, server WatchManager two tables, trigger-and-delete, notification without data; ② Properties — one-time, lightweight, ordering, disconnect invalidation; ③ Engineering — Curator's three caches automate re-registration and rebuild, business actions hang on ConnectionState; ④ Scale — herd effect and three-layer solution (parent→child, on-demand subscription, push-pull combined). Four parts cover mechanism, engineering, scale.

6.3 Twenty Interview Questions (with Answers)

Q1 · Complete flow of watch mechanism?

Answer: Client registers (getData/exists/children with watcher) → server WatchManager records tables → data change triggers FinalRequestProcessor reverse-lookup → constructs WatchedEvent with only type+path and pushes back → that watch removed from table → client EventThread callbacks. Notification lightweight, data must be read back.

Q2 · Why is watch designed as one-time?

Answer: Persistent watches mean server must maintain permanent entry per watcher and bear continuous change-matching pressure; client crashes make entry cleanup hard. One-time + callback re-registration shifts renewal responsibility to client, server implementation simple, leakage controllable. 3.6's persistent watch is the official solution after client capabilities caught up.

Q3 · Why doesn't watch notification carry data?

Answer: Separation of concerns: notification only signals "changed", data pulled by client on demand. Benefit: server push path light, different clients can read same change differently; cost: herd scenario traffic doubles (notification + read-back), needs granularity control to mitigate.

Q4 · What is watch's ordering guarantee?

Answer: Two guarantees: same client's watch events callback in order; client always sees watch event before seeing changed data. Therefore getData in callback always gets new value; Curator cache internal consistency relies on this.

Q5 · Difference between session expiry and connection disconnect?

Answer: Connection is TCP layer, session is server bookkeeping. After disconnect, session stays alive within sessionTimeout, reconnect restores; session expiry deletes ephemeral nodes, clears all watches, invalidates sessionId, must rebuild everything as new session. Curator maps to SUSPENDED (degrade) and LOST (rebuild).

Q6 · Are changes during disconnect re-sent?

Answer: No. Watch only triggers on the next change after registration; events in disconnect window are entirely lost (even the one-time trigger may be consumed during disconnect). So after reconnect you must actively do full pull reconciliation — Curator's rebuild is exactly this action.

Q7 · What does Curator solve over native client?

Answer: Four things: automatic watch re-registration and disconnect rebuild (three caches); connection state machine (ConnectionState, business can hook SUSPENDED/RECONNECTED/LOST); retry semantics (retryPolicy handles retryable exceptions); high-level recipes (locks, leader election, barriers, queues). Native API only for understanding principles.

Q8 · How to choose among NodeCache, PathChildrenCache, TreeCache?

Answer: Single node data changes → NodeCache (config hot-reload); directory direct children add/remove/update → PathChildrenCache (service discovery); entire subtree → TreeCache (hierarchical config, watch count grows with node count). Curator 5 unifies as CuratorCache.

Q9 · What is herd effect? How to solve?

Answer: One event triggers many clients to duplicate actions: N clients watch same node, one change produces N notifications + N read-backs. Solution in three layers: parent watch → child watch (shrink audience), on-demand subscription (only watch used paths), push-pull combined (local cache + periodic reconciliation safety net). Essence is watch granularity problem.

Q10 · How to observe herd effect?

Answer: Server four-letter commands: wchs for total, wchp by path to see which sessions watch it — single path with thousands of sessions is smoking gun; wchc by session to reverse-query. Combine with change frequency to estimate "notification volume per change", then compare wchs after fix.

Q11 · Can client callback directly query database?

Answer: Cannot do heavy work directly: EventThread (native) or cache's event thread is single-threaded; callback blocking queues all subsequent events, symptom is "notification latency grows". Callback only does light updates; heavy work goes to business thread pool.

Q12 · Can Curator's retryPolicy retry session expiry?

Answer: No. Retry only covers ConnectionLoss and similar idempotent retryable scenarios; SessionExpired means ephemeral nodes and watches are all gone, retrying same operation is meaningless. Correct handling: in LOST state rebuild ephemeral state (re-register, re-acquire lock).

Q13 · What should business do at SUSPENDED?

Answer: Pause writes to ZK and actions depending on strong consistency (lock renewal, registration changes); local cache can continue serving reads; wait for RECONNECTED to reconcile or LOST to rebuild. Simply: degrade without invalidation.

Q14 · Difference between exists and getData watch registration?

Answer: They hang on different tables: getData watch triggers NodeDataChanged/NodeDeleted; exists watch can also trigger NodeCreated (can watch non-existent node). Children triggers child list changes. This is why old "exists then get" pattern has race condition — Curator cache handles uniformly internally.

Q15 · Why does wchp show hundreds of watches on a path but no alert?

Answer: Watch entries themselves are light; hanging doesn't consume much resource; problem only appears when "that path changes". So alert on the product of two metrics: watch count × change frequency. Low-frequency path with hundreds of watches is acceptable; high-frequency path with dozens needs granularity fix.

Q16 · What is persistent watch (addWatch)?

Answer: 3.6 added resident watch: register once, stays effective, not deleted on trigger, supports PERSISTENT (this node) and PERSISTENT_RECURSIVE (subtree) modes. It officializes the manual "re-register in callback" action, prerequisite is client also upgraded to 3.6+.

Q17 · How to guarantee data doesn't change between notification and read-back?

Answer: When reading, you re-registered watch (getData with watch), so after read-back you're watching the new version; if two changes are extremely close, at most late by one event cycle, never permanently wrong. For absolute consistency, use stat's version for optimistic check on read-back; if mismatch, wait for next notification.

Q18 · How does server clean up dead connection watches?

Answer: On connection close (or session expiry), reverse-lookup all paths for that watcher via watch2Paths, batch-remove from watchTable; on trigger, skip push for already-closed connections (isClosed). Two paths guarantee no table leakage.

Q19 · Trade-offs: ZK as config center vs Nacos/Consul?

Answer: ZK strong consistency, lightweight watch, fits "config must take effect cluster-wide at same instant"; cost: write single point, election period unavailable, 1MB capacity limit. Nacos provides console, canary, config layering, productized ops, AP mode higher availability. Small scale + strong consistency → ZK; productized ops experience → Nacos.

Q20 · How do you prevent "zombie data after disconnect"?

Answer: Four-piece set: use Curator Cache not bare watch (auto rebuild); ConnectionStateListener hooks full state actions (SUSPENDED degrade, RECONNECTED reconcile, LOST rebuild); periodic full reconciliation safety net (30s level); local cache with TTL and version, expire without refresh → circuit-break degrade.

6.4 One-Page Cheat Sheet

Watch four properties: one-time · lightweight · ordering · disconnect invalidation Session ≠ connection: broken can return · expiry rebuild Reconnect first thing: rebuild reconciliation Three state gears: SUSPENDED degrade · RECONNECTED reconcile · LOST rebuild Herd three solutions: parent→child · on-demand subscribe · push-pull combined One iron law: callback no blocking (same as Netty)

VII. Summary

This article compresses to three sentences:

First, watch is a "one-time + no data" notification mechanism , server handles lightweight push; re-registration and reconciliation responsibility lies on client — Curator's entire value is automating these two responsibilities.

Second, session and connection are two lifecycles , disconnect recoverable, expiry requires rebuild; business actions must be mounted on SUSPENDED / RECONNECTED / LOST three gears.

Third, herd effect is a granularity problem , solution shrinks watch audience from "everyone" to "relevant ones", then uses push-pull combined as safety net.

Summary: With this, the three foundation blocks of the coordination kernel — "session, ephemeral nodes, watch" — are ready. They are precisely the raw materials for distributed locks and leader election: locks queue via ephemeral sequential nodes, leader election wins by smallest sequence number, safety net relies entirely on session expiry auto-cleanup. The next article "Distributed Lock & Leader Election Practice — Curator's Lock and LeaderLatch" assembles these three materials into truly production-ready primitives, clarifying herd lock vs fair lock trade-offs, classic lock loss accidents, and fencing token redemption.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Distributed SystemsService DiscoveryConfiguration ManagementZooKeeperSession ManagementCuratorWatch MechanismHerd Effect
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.