Designing High Availability for Canal Using Zookeeper: Distributed Locks and Watch Mechanism

We first introduce the problem: Canal servers subscribe to MySQL binlog for incremental data replication, and a single server failure would break the pipeline. To provide high availability (HA), we need a way to elect a primary server and let backups take over automatically.

We propose using Zookeeper (ZK) as a distributed coordination service. The primary server obtains a distributed lock by creating a unique node (e.g., /lock1 ) under the ZK root. The server that successfully creates the node becomes the primary, while others become backups.

Backups detect primary failure through ZK's watch mechanism. The primary maintains a long‑lived TCP connection and periodically sends heartbeats; if ZK does not receive a heartbeat within a timeout, it deletes the /lock1 node and notifies all watchers. Each backup registers itself in a mapping list under the same lock path, so only the relevant backups receive the notification.

To avoid the thundering‑herd problem when many backups try to acquire the lock simultaneously, we use sequential nodes. Each server creates a sequential child (e.g., sub-000001 , sub-000002 ) under /lock1 . The smallest sequence number holds the lock, and each server watches the node with the next lower sequence number, ensuring only one backup is awakened on failure.

This approach also provides a fair lock: lock acquisition order follows the creation order of sequential nodes, preventing starvation.

We then give a brief overview of Zookeeper concepts. ZK stores data in a tree‑like hierarchy of znodes, which can be temporary, persistent, sequential, or persistent‑sequential. Each znode can hold up to 1 MiB of data, making it suitable for configuration storage.

Watcher events allow clients to react to changes. ZK defines several event types, for example:

public enum EventType {
None (-1), // client connection state change
NodeCreated (1), // node creation event
NodeDeleted (2), // node deletion event
NodeDataChanged (3), // node data change
NodeChildrenChanged (4); // child node added or removed
}

Typical ZK applications include:

Service registry (e.g., Dubbo): services register temporary nodes under a path; clients watch NodeChildrenChanged to discover live instances.

Configuration center: services watch NodeDataChanged on config nodes to receive real‑time updates, as used by the QConf system.

In summary, understanding Zookeeper's tree structure, node types, and watch mechanism enables building HA solutions, distributed locks, naming services, and configuration management. For production use, a ZK ensemble is required to avoid single‑point failure, and deeper topics such as the ZAB protocol and leader election should be studied.