How to Seamlessly Migrate Zookeeper for ClickHouse Across Data Centers with Zero Downtime

Background

The ClickHouse (ck) cluster was already moved from Hong Kong to Singapore, but its dependent Zookeeper (zk) cluster remained in Hong Kong. The goal was to migrate zk to Singapore without affecting ck’s real‑time analytics workload.

1. Objectives and Challenges

Ensure zk migration is transparent to ck users; ck must remain fully available.

Perform hot upgrades and dynamic configuration updates to minimise zk downtime.

2. Overall Solution

2.1 Upgrade zk from static to dynamic configuration

zk 3.5+ supports dynamic configuration, which simplifies scaling. The existing zk cluster used the static version, so the first step was a smooth upgrade to the dynamic version.

Reference: https://backendhouse.github.io/post/zookeeper_dynamic_config/

2.2 Expand/contract zk to move machines

Expand: add new Singapore machines one by one.

Update ck configuration to point to the new machines.

Shrink: remove old Hong Kong machines one by one.

3. Problems Encountered and Solutions

3.1 Incompatibility between static‑version Leader and dynamic‑version Follower

During serial upgrade, Followers upgraded to dynamic config reported errors because the static Leader rejected a higher config version:

2021-02-25 11:07:03,081 [myid:5] - WARN  [QuorumPeer[myid=5]] - Exception when following the leader
java.io.EOFException
... (stack trace) ...
java.lang.Exception: shutdown Follower

The Leader (still static) rejected the Follower’s newer config version, causing "Follower is ahead of the leader" exceptions.

Two possible fixes were considered:

Switch to parallel upgrade (risky due to 2‑4 min zk outage).

Introduce a temporary static version that disables the config‑version check, then upgrade to the true dynamic version.

Because parallel upgrade introduced unacceptable risk, the team chose the second approach: first upgrade to a static version without config‑version verification, then to the dynamic version.

3.2 ClickHouse cannot dynamically reload zk server list

Changing the zk server list in ck’s configuration did not trigger a live reload, requiring a full ck restart, which is unacceptable for many production services.

Solution: add dynamic zk‑config loading support to ClickHouse (merged upstream). PR: https://github.com/ClickHouse/ClickHouse/pull/14678

3.3 zk restart caused occasional ck query failures

When the optimize_trivial_count_query flag is enabled, simple SELECT count(*) queries read metadata from zk. During zk migration these queries failed.

Fix: disable the flag before migration and re‑enable it after the zk cluster is stable.

3.4 zk restart caused ck write failures

ck relies on zk for block‑id allocation and replica synchronization. During zk downtime writes may fail, so retry mechanisms were added to all data‑ingestion pipelines (Flink, Spark, clickhouse_sinker).

3.5 zk restart caused ck table‑change failures

Table DDL operations also touch zk. To avoid failures, the team temporarily disabled DDL during the migration.

4. Final Migration Plan

4.1 Initial State

Hong Kong machines: A1‑A5. Singapore machines: B1‑B5. zk runs on the static version in Hong Kong.

4.2 Upgrade to Dynamic Configuration

Upgrade path: static → static (no config‑version check) → dynamic.

Steps (serial upgrade):

Upgrade Followers first; each upgrade may cause brief ck write errors (session expired) that recover within ~40 s.

Upgrade Leader last; election may cause ck writes to fail with "table in readonly mode" for up to 3 min.

Rollback: parallel rollback to the original static version.

4.3 Dynamic Expansion (Add Singapore nodes)

Serially add each new node (B1‑B5) using reconfig -add, verify configuration propagation, ck connectivity, and absence of errors before proceeding to the next node.

Impact: none expected.

4.4 Update ClickHouse zk configuration

Replace <zookeeper-servers> entries from A1‑A5 with B1‑B5 and redeploy to all ck instances. Verify connections with netstat.

<zookeeper-servers>
  <node index="0"><host>A1</host><port>2181</port></node>
  <node index="1"><host>A2</host><port>2181</port></node>
  ...
</zookeeper-servers>

Impact: none expected.

4.5 Shrink (Remove Hong Kong nodes)

Serially remove each old node (A1‑A5) using reconfig -remove, checking zk status and ck read/write health after each removal.

Impact: same as in the upgrade steps (brief write interruptions, leader election delays).

5. Conclusion

Extensive offline rehearsals identified and resolved all migration‑related issues. By combining hot upgrades, dynamic configuration, careful expansion, configuration reload, and staged shrinkage, the zk cluster was moved from Hong Kong to Singapore with virtually no impact on ClickHouse users.