How to Safely Upgrade a ZooKeeper Node’s IP Without Disrupting the Cluster

Background and Principle

ZooKeeper nodes communicate via the server.X=ip:port:port entries in zoo.cfg and identify themselves with a myid file. When a node’s IP changes, other members can no longer reach it, and the cluster treats the node as offline because the IP in the configuration no longer matches.

Therefore, every node’s configuration must be updated to the new IP before the upgraded node can re‑join the ensemble.

Step‑by‑Step Upgrade Procedure

Assumed original three‑node cluster

server.1=192.168.1.101:2888:3888
server.2=192.168.1.102:2888:3888
server.3=192.168.1.103:2888:3888

We will replace server.3 with the new IP 192.168.1.203.

Step 1 – Stop the node to be upgraded

zkServer.sh stop

Verify that the process has stopped.

Step 2 – Update zoo.cfg on all nodes

On each server (1, 2, and 3) replace the old line: server.3=192.168.1.103:2888:3888 with the new line: server.3=192.168.1.203:2888:3888 Note: Do not change the myid file (it remains 3) and keep the ports 2888/3888 unchanged.

Step 3 – Restart the remaining nodes (if they were not stopped)

zkServer.sh restart

This forces the new configuration to take effect.

Step 4 – Start the upgraded node

zkServer.sh start

Step 5 – Verify the cluster

Run on any node: zkServer.sh status You should see one Leader and two Followers (including the newly‑upgraded node).

Important Considerations

Changing IPs in a running ZooKeeper ensemble is not recommended because ZooKeeper does not support dynamic discovery; all nodes must load the new configuration simultaneously.

In production, perform the upgrade in two phases: (1) take the target node offline, edit its config, and bring it back up; (2) restart the other nodes.

When deploying on Kubernetes, avoid hard‑coding IPs. Use a StatefulSet with a headless Service so each pod gets a stable DNS name (e.g., zk-0.zk-hs.default.svc.cluster.local), and reference these DNS names in zoo.cfg instead of IPs.

Kubernetes‑Native ZooKeeper Deployment (IP‑Free)

Typical manifests:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zk
spec:
  serviceName: zk-hs
  replicas: 3
  selector:
    matchLabels:
      app: zk
  template:
    metadata:
      labels:
        app: zk
    spec:
      containers:
      - name: zookeeper
        image: zookeeper:3.8
        ports:
        - containerPort: 2181
        - containerPort: 2888
        - containerPort: 3888

apiVersion: v1
kind: Service
metadata:
  name: zk-hs
spec:
  clusterIP: None
  selector:
    app: zk
  ports:
  - port: 2181
    name: client
  - port: 2888
    name: follower
  - port: 3888
    name: election

In zoo.cfg use DNS names instead of IPs:

server.1=zk-0.zk-hs.default.svc.cluster.local:2888:3888
server.2=zk-1.zk-hs.default.svc.cluster.local:2888:3888
server.3=zk-2.zk-hs.default.svc.cluster.local:2888:3888

With this setup, pod IP changes or node migrations no longer require manual configuration updates, simplifying upgrades and improving reliability.