Upgrade a Kubernetes Cluster from v1.22 to v1.23 the Hard Way

Kubernetes "hard‑way" refers to performing each upgrade step manually to understand the underlying processes, which builds troubleshooting skills and gives fine‑grained control over the cluster.

Prerequisites

Back up all critical components, especially application state stored in databases; upgrades affect only Kubernetes components.

Disable swap.

Master Node Upgrade

Drain the master node, ignoring daemon‑set pods: #kubectl drain master --ignore-daemonsets Verify the node is SchedulingDisabled with #kubectl get nodes.

Update the package manager (e.g., #apt-get update).

Search for the desired kubeadm version (1.23) and note the package name.

Hold kubectl and kubelet to prevent automatic changes: #apt-mark hold kubectl kubelet Unhold kubeadm, install the specific version, then hold it again:

#apt-mark unhold kubeadm
#apt-get install kubeadm=1.23.1-00
#apt-mark hold kubeadm

Confirm the kubeadm version: #kubeadm version View the upgrade plan: #kubeadm upgrade plan Apply the upgrade: #kubeadm upgrade apply v1.23.1 Unhold and upgrade kubectl and kubelet, then hold them again:

#apt-mark unhold kubelet kubectl
#apt-get install kubelet=1.23.1-00 kubectl=1.23.1-00
#apt-mark hold kubelet kubectl

Restart kubelet and check its status:

#systemctl restart kubelet
#systemctl status kubelet

Run #kubeadm upgrade plan again to verify.

Check node status and ensure the master reports the new version.

Uncordon the master to make it schedulable again:

#kubectl uncordon master
#kubectl get nodes

Worker Node Upgrade

Drain the worker node, forcing pod eviction if needed:

#kubectl drain worker --ignore-daemonsets --force

Update the package manager, unhold kubeadm, install the new version, and hold it again:

#apt-get update
#apt-mark unhold kubeadm
#apt-get install kubeadm=1.23.1-00
#apt-mark hold kubeadm

Unhold and upgrade kubectl and kubelet, then hold them:

#apt-mark unhold kubelet kubectl
#apt-get install kubelet=1.23.1-00 kubectl=1.23.1-00
#apt-mark hold kubelet kubectl

Restart kubelet and verify its status:

#systemctl restart kubelet
#systemctl status kubelet

Check node status to confirm the worker reports the new version.

Uncordon the worker node to make it schedulable again:

#kubectl uncordon worker
#kubectl get nodes

After completing these steps, the Kubernetes cluster is successfully upgraded from v1.22.2 to v1.23.1, with all components aligned to the same version.

Conclusion

We upgraded the cluster version, the kubeadm package, and synchronized kubectl and kubelet to version 1.23.1. The core workflow involves draining nodes, unholding packages, performing the upgrade, re‑holding packages, and finally uncordoning the nodes.