How to Upgrade a High‑Availability Kubernetes Cluster and Etcd with Zero Downtime

Kubernetes deployment is complex because it bundles many components, and keeping a high‑availability (HA) cluster up‑to‑date requires coordinated upgrades of both the control plane and the etcd datastore.

Upgrade Kubernetes

The example assumes a cluster with three master nodes (v1.13) and one worker node.

1. Upgrade kubeadm on each master

$ ssh [email protected]
$ apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.13.0-00 && apt-mark hold kubeadm

Holding the package prevents the installer from automatically upgrading dependent components such as kubelet.

2. Verify the upgrade plan

$ kubeadm upgrade plan
... (output shows current and available versions for API Server, Controller Manager, Scheduler, Kube‑Proxy) ...

3. Apply the upgrade plan

$ kubeadm upgrade apply v1.14.0

4. Update kubelet and restart the service

$ apt-mark unhold kubelet && apt-get update && apt-get install -y kubelet=1.14.0-00 && apt-mark hold kubelet
$ systemctl restart kubelet

5. Upgrade the remaining master nodes

$ ssh [email protected]
$ kubeadm upgrade node experimental-control-plane
$ ssh [email protected]
$ kubeadm upgrade node experimental-control-plane

6. Upgrade kubectl on all masters

$ apt-mark unhold kubectl && apt-get update && apt-get install -y kubectl=1.14.0-00 && apt-mark hold kubectl

7. Drain, upgrade, and restore the worker node

$ ssh [email protected]
$ kubectl drain worker --ignore-daemonsets
$ apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.14.0-00 && apt-mark hold kubeadm
$ apt-mark unhold kubelet && apt-get update && apt-get install -y kubelet=1.14.0-00 && apt-mark hold kubelet
$ systemctl restart kubelet
$ kubectl uncordon worker

Before each step, perform regular snapshots and health checks to ensure a safe rollback path.

Upgrade Etcd

Etcd is the distributed key‑value store that backs Kubernetes. An HA setup typically runs at least three etcd nodes, and upgrades must be performed one version at a time.

Preparation

Check the current etcd version: $ ./etcdctl endpoint status Backup the data directory before making any changes.

Upgrade Procedure

Log into the first etcd node and stop the running process:

$ ssh 10.0.1.1
$ kill $(pgrep etcd)

Backup the etcd data:

$ ./etcdctl backup --data-dir %data_dir% --backup-dir %backup_data_dir%

Download the new etcd binaries and start the server with the same configuration:

ETCD_VER=v3.3.15
# Choose a download URL (Google or GitHub)
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /usr/local/etcd && mkdir -p /usr/local/etcd
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /usr/local/etcd --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
/usr/local/etcd/etcd --version
ETCDCTL_API=3 /usr/local/etcd/etcdctl version
/usr/local/etcd/etcd -name etcd-1 -listen-peer-urls http://10.0.1.1:2380 -listen-client-urls http://10.0.1.1:2379,http://127.0.0.1:2379 -advertise-client-urls http://10.0.1.1:2379,http://127.0.0.1:2379

Repeat steps 1‑3 on the remaining etcd nodes.

Verify cluster health: $ ./etcdctl endpoint health If TLS is enabled, provide the certificate files or set the environment variables ETCD_CA_FILE, ETCD_CERT_FILE, and ETCD_KEY_FILE for the commands.

Version Compatibility

Kubernetes v1.0 – etcd2 only

Kubernetes v1.5.1 – adds etcd3 support (new clusters default to etcd2)

Kubernetes v1.6.0 – new clusters default to etcd3

Kubernetes v1.9.0 – etcd2 deprecated

Kubernetes v1.13.0 – etcd2 removed; clusters must use etcd3

Therefore, when upgrading a cluster from v1.12 (which may still use etcd2) to v1.13, you must also upgrade etcd to a v3 release.