How to Diagnose and Fix Expired Kubernetes Certificates with kubeadm

1. Typical failures after certificate expiration

Common symptoms include:

kubectl cannot access the cluster – error x509: certificate has expired Node NotReady – kubelet logs report certificate issues

Core control‑plane services unavailable – apiserver, controller‑manager, scheduler, etcd cannot load expired certificates

Conclusion: expired certificates can cripple the entire control plane, so renewal must occur at least 30 days before expiry.

2. Complete certificate renewal workflow (production ready)

Core principle: backup before operating. Run during a low‑traffic window.

Step 1: Check current certificate expiration

kubeadm certs check-expiration

Focus on the EXPIRES field and RESIDUAL TIME ; if less than 30 days, renewal is required.

Step 2: Backup critical directories

Directories to back up : /etc/kubernetes – control‑plane configuration and kubeconfig /etc/kubernetes/pki – all core certificates and keys /var/lib/kubelet/pki – kubelet client certificates $HOME/.kube/config – administrator kubeconfig /etc/kubernetes/manifests – static Pods

Recommended backup commands :

sudo cp -r /etc/kubernetes /etc/kubernetes.bak_$(date +%Y%m%d)
sudo cp -r /var/lib/kubelet/pki /var/lib/kubelet/pki.bak_$(date +%Y%m%d)
cp ~/.kube/config ~/.kube/config.bak.$(date +%F)

Step 3: Renew master node certificates

For HA clusters, repeat on each master.

1. Renew all certificates sudo kubeadm certs renew all This re‑signs certificates with the cluster CA, extending validity by one year.

2. Restart control‑plane components sudo systemctl restart kubelet Wait 2–3 minutes, then verify that the following components are Running in the kube-system namespace:

kube-apiserver

kube-controller-manager

kube-scheduler

etcd

3. Update local kubeconfig

sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Step 4: Handle worker node certificates (auto‑rotation & manual fix)

4.1 Verify auto‑rotation is enabled

cat /var/lib/kubelet/config.yaml | grep rotateCertificates
ps aux | grep kubelet | grep rotate

Expected output includes rotateCertificates: true and the --rotate-certificates flag.

4.2 Approve pending CSRs

kubectl get csr
kubectl get csr -o name | xargs kubectl certificate approve

4.3 Manually recover completely expired kubelet certificates

On the affected worker node:

sudo systemctl stop kubelet
sudo rm -rf /var/lib/kubelet/pki/kubelet-client-*
sudo systemctl start kubelet

Then on the master, approve the new CSR:

kubectl get csr -o name | xargs kubectl certificate approve

Step 5: Post‑renewal verification

Check certificate expiration again kubeadm certs check-expiration Node health check kubectl get nodes All nodes should show Ready.

System pod health check kubectl get pods -n kube-system Test cluster usability

kubectl run test --image=busybox --restart=Never -- sleep 5

Long‑term maintenance recommendations

Run quarterly

kubeadm certs check-expiration

Monitoring & alerting (Prometheus/Alertmanager)

Remaining days < 60: warning

Remaining days < 30: critical alert

Cross‑machine backups

Automate daily backup of:

/etc/kubernetes

/etc/kubernetes/pki