Kubernetes Certificate Expiration Demystified: Incident Postmortem & 11‑Step Renewal Guide

Incident Overview

A payment‑platform running a kubeadm v1.26.x cluster (3 masters, 24 workers, 200+ micro‑services) experienced a cascade of failures when the control‑plane certificates expired. The API server became unreachable, pods could not be scheduled, nodes turned NotReady, CoreDNS failed, and CI/CD pipelines and monitoring lost cluster access.

Root Causes

No certificate‑expiration monitoring.

No runbook for control‑plane renewal.

No etcd snapshot and recovery drills.

No disciplined rolling maintenance for multi‑master clusters.

Certificate Landscape

Kubernetes uses three PKI authorities: ca – the cluster root CA, signs API server, component kubeconfigs and kubelet client certificates. etcd/ca – signs etcd server, peer and health‑check certificates. front‑proxy‑ca – signs the front‑proxy client used by aggregated APIs.

When any non‑CA certificate expires, both the server identity and the client identity that depend on it become invalid, breaking the whole trust chain.

Impact of Specific Expired Certificates

apiserver.crt

– API server health checks fail, load balancer removes the endpoint. admin.conf – CI/CD, monitoring and any external tool using the embedded client certificate lose access. apiserver‑etcd‑client.crt – API server cannot read/write etcd. etcd/server.crt or etcd/peer.crt – etcd quorum may be lost. kubelet.conf – nodes become NotReady and stop reporting pod status.

11‑Step Renewal Procedure for kubeadm Multi‑Master Clusters

Stop loss : Verify health endpoints before taking any action.

curl -k https://127.0.0.1:6443/healthz
curl -k https://127.0.0.1:6443/livez
curl -k https://127.0.0.1:6443/readyz

Backup PKI and etcd :

#!/usr/bin/env bash
set -euo pipefail
BACKUP_BASE="/backup"
TS=$(date +%Y%m%d%H%M%S)
BACKUP_DIR="${BACKUP_BASE}/k8s-cert-backup-${TS}"
mkdir -p "${BACKUP_DIR}"
cp -a /etc/kubernetes "${BACKUP_DIR}/"
cp -a /var/lib/kubelet/pki "${BACKUP_DIR}/kubelet-pki" || true
cp -a /etc/systemd/system/kubelet.service.d "${BACKUP_DIR}/kubelet-systemd" || true
if systemctl is-active --quiet etcd; then
  ETCDCTL_API=3 etcdctl \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
    --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
    snapshot save "${BACKUP_DIR}/etcd-snapshot.db" || true
fi
cp -a /var/lib/etcd "${BACKUP_DIR}/var-lib-etcd" || true
tar czf "${BACKUP_DIR}.tar.gz" -C "${BACKUP_BASE}" "$(basename "${BACKUP_DIR}")"
echo "backup saved: ${BACKUP_DIR}.tar.gz"

Both an etcd logical snapshot and a copy of /var/lib/etcd are kept – the snapshot is used for normal restores, the raw directory is useful for forensic analysis.

Identify expired certificates and CA validity :

kubeadm certs check-expiration
openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -dates
openssl x509 -in /etc/kubernetes/pki/etcd/ca.crt -noout -dates
openssl x509 -in /etc/kubernetes/pki/front-proxy-ca.crt -noout -dates

Determine whether the expired objects are leaf certificates or the CA itself.

Confirm cluster topology and rolling order List master nodes, their IPs/hostnames, load‑balancer back‑ends and etcd member list ( etcdctl member list ).

Renew control‑plane certificates (full or component‑by‑component):

# renew everything
kubeadm certs renew all
# or renew selectively
kubeadm certs renew apiserver
kubeadm certs renew apiserver-etcd-client
kubeadm certs renew controller-manager.conf
kubeadm certs renew scheduler.conf
kubeadm certs renew admin.conf

kubeadm certs renew reads the local CA key and re‑signs the target certificates, overwriting the original files. It does **not** restart any component.

Update kubeconfig files used by external systems :

mkdir -p ~/.kube
cp /etc/kubernetes/admin.conf ~/.kube/config
chown $(id -u):$(id -g) ~/.kube/config

Inspect the embedded client certificate in admin.conf to ensure it is refreshed:

kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d | openssl x509 -noout -subject -dates

All external tools (GitLab Runner, Argo CD, monitoring scripts, etc.) must be pointed to the new kubeconfig.

Rolling restart of static Pods (move manifest away and back so kubelet recreates the pod):

mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
sleep 20
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/
# repeat for controller‑manager and scheduler

Verify each component after restart (e.g., curl -k https://127.0.0.1:6443/healthz , kubectl get pods -n kube-system ).

If etcd certificates are also expired, renew them first and restart etcd :

kubeadm certs renew etcd-server
kubeadm certs renew etcd-peer
kubeadm certs renew etcd-healthcheck-client
kubeadm certs renew apiserver-etcd-client
mv /etc/kubernetes/manifests/etcd.yaml /tmp/
sleep 20
mv /tmp/etcd.yaml /etc/kubernetes/manifests/
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  endpoint health

Refresh kubelet client certificates on workers :

# generate a fresh join token (valid for 2 h)
kubeadm token create --ttl 2h --print-join-command
# on each worker run the join‑phase to obtain a new kubelet certificate
kubeadm join-phase kubelet-start <em>API_SERVER_LB</em>:6443 \
  --token <em>TOKEN</em> \
  --discovery-token-ca-cert-hash sha256:<em>HASH</em>
systemctl restart kubelet

Do not copy the master’s kubelet.conf to workers – each node has its own identity.

End‑to‑end validation :

# certificate dates
kubeadm certs check-expiration
# node health
kubectl get nodes -o wide
# control‑plane pods
kubectl get pods -n kube-system -o wide | grep -E 'apiserver|etcd|controller-manager|scheduler'
# DNS
kubectl run dns-test --rm -it --image=busybox --restart=Never -- nslookup kubernetes.default
# business workload sanity
kubectl run test-deploy --image=nginx --restart=Never

Long‑term governance

Monitor certificate expiry (e.g., Prometheus metric k8s_cert_expiry_days).

Schedule regular etcd snapshots and half‑yearly recovery drills.

Automate renewal with a lock, threshold check and staged restarts (sample script shown in the article).

Centralise external kubeconfig management and enforce RBAC for CSR auto‑approval.

Keep leaf certificates short‑lived (≈1 year) and the root CA long‑lived (≈10 years).

Common Pitfalls

Running kubeadm certs renew all without restarting static Pods leaves the old certificates in use.

Restarting all masters simultaneously breaks HA.

Deleting /var/lib/etcd without a snapshot destroys cluster state.

Updating only admin.conf does not refresh external kubeconfigs.

“Not yet valid” errors are often caused by clock drift – verify NTP.

Extending leaf‑certificate lifetimes does not replace a proper renewal process.

CA Expiration Handling

If step 3 reveals that the cluster CA itself is expired, kubeadm certs renew cannot help because it needs the CA private key to sign new certificates. In that situation the recovery path changes:

Back up /var/lib/etcd and the entire /etc/kubernetes/pki directory.

Choose one of two strategies:

Strategy A – Preserve etcd, rebuild control plane : generate a new CA, re‑initialize the control plane with the new CA, then restore the etcd data snapshot.

Strategy B – Re‑create the whole cluster : stand up a fresh cluster and migrate workloads; useful when the existing PKI is unrecoverable.

Avoid destructive actions such as deleting /var/lib/etcd unless you have a verified snapshot.

Engineering Automation

Simple cron jobs that blindly execute kubeadm certs renew all are insufficient. Production‑grade automation must include:

Mutual‑exclusion lock so only one master performs renewal at a time.

Threshold check (e.g., renew when k8s_cert_expiry_days < 45).

Stage‑wise renewal and restart (etcd → apiserver → controller‑manager → scheduler).

Health verification after each stage; abort on failure.

Auditable logging of every step.

Example script (truncated for brevity):

#!/usr/bin/env bash
set -euo pipefail
LOCK_FILE="/var/lock/k8s-cert-renew.lock"
THRESHOLD=45
exec 200>"${LOCK_FILE}" || exit 1
flock -n 200 || { echo "another renew task is running"; exit 0; }
LEFT=$(days_left /etc/kubernetes/pki/apiserver.crt)
if [ "$LEFT" -gt "$THRESHOLD" ]; then echo "no renewal needed"; exit 0; fi
kubeadm certs renew all
restart_static_pod etcd
# verify etcd health, abort if failed
restart_static_pod kube-apiserver
# verify apiserver health, abort if failed
restart_static_pod kube-controller-manager
restart_static_pod kube-scheduler
echo "renewal completed"

Kubelet Automatic Rotation

Even when rotateCertificates: true and serverTLSBootstrap: true are set, kubelet rotation can fail for three reasons:

Bootstrap kubeconfig is already invalid.

CSR auto‑approval RBAC is missing or mis‑configured.

API server was down during the rotation window.

Ensure the following:

Kubelet rotation is enabled and verified in a non‑production window.

Cluster has a ClusterRoleBinding that auto‑approves node client CSRs (see article for the exact manifest).

Clock synchronization (NTP) is healthy.

Business‑Level Impact Example

In the payment‑order platform the load during a promotion reached >18 000 QPS, Kafka ingest >1.2 M messages/min, and >1 500 pods. Certificate expiry caused:

CI/CD pipelines to stop, preventing urgent hot‑fix releases.

Horizontal pod autoscaler to freeze, leading to node overload.

New pod creation to fail, causing a chain‑reaction of capacity exhaustion.

Monitoring loss, making root‑cause analysis harder.

Runbook Snapshot (Condensed)

# Fault detection
kubectl get nodes
kubeadm certs check-expiration
curl -k https://127.0.0.1:6443/readyz

# Backup
cp -a /etc/kubernetes /backup/
cp -a /var/lib/etcd /backup/

# Renew
kubeadm certs renew all

# Rolling restart order
restart_static_pod etcd
restart_static_pod kube-apiserver
restart_static_pod kube-controller-manager
restart_static_pod kube-scheduler

# Verify
kubectl get --raw='/readyz?verbose'
kubectl get nodes
kubectl get pods -A

# Refresh external systems (CI/CD, monitoring, GitOps)
# (copy updated admin.conf or distribute new kubeconfig)

Decision Guidance: Self‑Managed vs Managed

Self‑managed kubeadm offers full flexibility but requires you to own the entire certificate lifecycle, etcd backup/recovery, and HA rolling procedures.

Self‑managed + platform governance adds a central automation layer (monitoring, runbooks, audit) that reduces human error while keeping the ability to customise the stack.

Managed Kubernetes services offload control‑plane maintenance, but you still need to manage node‑level certificates, CSR approval and external kubeconfig rotation.

Final Takeaways

The core problem is a broken control‑plane identity trust chain . Recovery is not a single renew all command; it requires:

Backing up PKI and etcd.

Ensuring the CA is still valid (or rebuilding it).

Renewing leaf certificates in dependency order (etcd → API server → controllers → kubelet).

Rolling restart of static Pods with strict single‑master discipline.

Updating every external kubeconfig and verifying kubelet rotation.

Embedding monitoring, alerting, audit and regular recovery drills into the operational process.

If you only remember one action, make it “check the remaining days of every certificate, back up etcd, and run the 11‑step renewal with rolling restarts” . This transforms a reactive fire‑fighting episode into a repeatable, auditable, and observable process.