Practical kubeadm Certificate Renewal: PKI Basics, HA Architecture, Automation

Conclusion: Certificate renewal is more than a single command

Many teams think renewing kubeadm certificates is just kubeadm certs renew all, but the real process involves understanding the PKI topology, updating kubeconfigs, performing rolling restarts, backing up files and etcd snapshots, and handling HA-specific considerations.

Do you understand the kubeadm control‑plane PKI topology and which certificates can be auto‑renewed?

Do you know why kubectl may still fail after renewal?

Can you roll out updates node‑by‑node in a multi‑control‑plane cluster with rollback capability?

Do you have pre‑alerting, automatic backup, failure circuit‑breaker, and post‑audit mechanisms?

Have you elevated this from a manual operation to a platform governance capability?

If any answer is no, certificate renewal remains a high‑risk manual task rather than a reliable engineering system.

Real background: why certificates expire during busy periods

Typical production symptoms when a control‑plane certificate expires: kubectl get nodes returns x509: certificate has expired or is not yet valid kube‑apiserver cannot authenticate kubelet or controller requests

kube‑controller‑manager / kube‑scheduler fail because embedded client certificates are invalid

etcd and apiserver TLS communication breaks

Service becomes unreachable, Ingress returns many 5xx, DNS resolution glitches, and service discovery jitter

Because the control‑plane relies almost entirely on TLS, an expired certificate means the whole control‑plane loses its trust relationship, making certificate renewal a critical part of cluster lifecycle management.

PKI topology

Cluster Root CA
├── apiserver.crt / apiserver.key
├── apiserver-kubelet-client.crt / key
├── controller-manager.conf (embedded client cert)
├── scheduler.conf (embedded client cert)
├── admin.conf (embedded client cert)
├── kubelet.conf (may be rotated)
├── front-proxy-ca.crt / key
│   └── front-proxy-client.crt / key
└── etcd-ca.crt / key
    ├── etcd/server.crt / key
    ├── etcd/peer.crt / key
    ├── etcd/healthcheck-client.crt / key
    └── apiserver-etcd-client.crt / key

ServiceAccount KeyPair
├── sa.key
└── sa.pub

Three easy‑to‑confuse points: /etc/kubernetes/pki/*.crt/*.key are the raw certificate and private‑key files. /etc/kubernetes/*.conf are kubeconfig files that may embed base64‑encoded client certificates. sa.key/sa.pub are ServiceAccount signing keys, not X.509 certificates.

After running kubeadm certs renew all, the pki directory contains new certificates, but kubectl often still fails because the embedded client certificates inside admin.conf, controller-manager.conf, and scheduler.conf have not been refreshed.

What kubeadm certs renew actually does

Read the existing CA / front‑proxy CA / etcd CA private keys.

Re‑sign subordinate certificates using the original subject‑alternative‑names.

It does NOT automatically hot‑load the new certificates, orchestrate a rolling update of control‑plane nodes, update all kubeconfig users, create backups, or manage change windows.

Why restart after renewal

Control‑plane components run as static Pods whose manifests live under /etc/kubernetes/manifests/. The components read certificates at start‑up and keep them in memory or hold open file handles, so replacing the files on disk does not trigger a hot reload.

/etc/kubernetes/manifests/etcd.yaml
/etc/kubernetes/manifests/kube-apiserver.yaml
/etc/kubernetes/manifests/kube-controller-manager.yaml
/etc/kubernetes/manifests/kube-scheduler.yaml

The correct update flow is:

Generate new certificates → Update kubeconfig files → Restart relevant components → Components re‑mount and read the new certificates → Health checks pass

Therefore, executing only kubeadm certs renew all is insufficient.

Single‑control‑plane vs HA control‑plane

Single control‑plane

Simple to implement, low maintenance cost.

During renewal the control‑plane has a noticeable risk window.

Suitable for test environments, low‑risk internal services, or small edge clusters.

HA control‑plane

Allows per‑node rolling updates.

More suitable for production.

Operational complexity is higher.

Applicable to core‑business production clusters, multi‑tenant platforms, or any scenario requiring high control‑plane availability.

Production‑grade runbook

Pre‑check

kubeadm certs check-expiration
kubectl get nodes -o wide
kubectl -n kube-system get pods -o wide
etcdctl endpoint health --cluster

Ensure the control‑plane is healthy, etcd has no member issues, kubelet is not in a prolonged NotReady state, and the renewal is not being performed during another incident.

Backup

Copy /etc/kubernetes/pki Copy /etc/kubernetes/*.conf Copy /etc/kubernetes/manifests Take an etcd snapshot

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  --endpoints=https://127.0.0.1:2379 \
  snapshot save /backup/etcd-snapshot-$(date +%Y%m%d%H%M%S).db

Renew

kubeadm certs renew all

Or renew per component, e.g. kubeadm certs renew apiserver, kubeadm certs renew front-proxy-client, etc.

Refresh kubeconfig

kubeadm init phase kubeconfig admin
kubeadm init phase kubeconfig controller-manager
kubeadm init phase kubeconfig scheduler
kubeadm init phase kubeconfig kubelet --node-name "$(hostname)"

If this step is skipped, kubectl continues to use the old client certificates and the control‑plane components may fail to start.

Restart static Pods in order

etcd → kube-apiserver → kube-controller-manager → kube-scheduler → kubelet

The order matters because apiserver depends on etcd, and the controller‑manager and scheduler depend on a healthy apiserver.

Verification

File layer: confirm certificate expiration dates have been refreshed.

Process layer: ensure static Pods are Running.

Interface layer: curl -k https://127.0.0.1:6443/healthz returns OK.

Cluster layer: kubectl get nodes/pods shows normal status.

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
curl -k https://127.0.0.1:6443/healthz
KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes
KUBECONFIG=/etc/kubernetes/admin.conf kubectl -n kube-system get pods

Why many tutorials break on HA clusters

Missing load‑balancer drain

Before updating a control‑plane node, remove it from the LB backend to avoid traffic hitting a node that is being restarted.

Missing per‑node certificate copies

In HA setups each control‑plane node maintains its own /etc/kubernetes/pki and *.conf. Running the renewal on only one master leaves the others with stale certificates.

Missing etcd peer consistency

When using stacked etcd, each node also has etcd/server.crt, etcd/peer.crt, and etcd/healthcheck-client.crt. Inconsistent renewal can cause TLS handshake failures, leader election jitter, or apiserver‑etcd communication loss.

The real difficulty in HA is not the renew command itself but performing a coordinated, batch‑wise switch of control‑plane and etcd certificates.

Production‑grade architecture: platform‑level certificate governance

A platform that turns certificate renewal into a governance capability typically consists of:

Prometheus / Alertmanager
    |
    v
Certificate Governance Controller
    |
    +-- Inventory & Expiry Scanner
    +-- Renewal Orchestrator
    +-- Backup Executor
    +-- LB Drain / Recover Adapter
    +-- Validation & Rollback Engine
    +-- Audit Event Publisher

Key capabilities

Certificate scanner: periodically scans control‑plane certificates, emits metrics, and flags high‑risk nodes.

Update orchestrator: decides when a renewal window is allowed, triggers per‑node renewal, and controls batch concurrency.

Backup executor: performs file‑level backup and etcd snapshot with traceable change IDs.

Validation & rollback engine: checks component health, aborts further batches on failure, and can roll back files or etcd data.

Audit & notification: publishes alerts to Prometheus, Alertmanager, and chat systems (e.g., Slack, Feishu).

Why this is a high‑concurrency, scalable problem

Scanning dozens to hundreds of clusters is concurrent.

Batch scheduling is needed when many clusters trigger renewal at the same time (e.g., monthly maintenance windows).

Global mutual exclusion is required to avoid simultaneous updates on the same cluster.

The solution must support single‑control‑plane, HA, stacked etcd, external etcd, and various execution methods (SSH, DaemonSet, platform job system).

Production‑grade shell script

#!/usr/bin/env bash
# kubeadm-cert-rotate.sh
set -Eeuo pipefail

DRY_RUN=false
RENEW_THRESHOLD_DAYS=90
BACKUP_ROOT="/var/backups/kubeadm-certs"
NODE_NAME="$(hostname -s)"
TIMESTAMP="$(date +%Y%m%d%H%M%S)"
BACKUP_DIR="${BACKUP_ROOT}/${NODE_NAME}-${TIMESTAMP}"
LOCK_FILE="/var/run/kubeadm-cert-rotate.lock"
KUBECONFIG_PATH="/etc/kubernetes/admin.conf"

log() {
  local level="$1"; shift
  printf '[%s] [%s] %s
' "$(date '+%F %T')" "$level" "$*"
}

fail() { log "ERROR" "$*"; exit 1; }

cleanup() { rm -f "${LOCK_FILE}"; }
trap cleanup EXIT
trap 'fail "Script interrupted, check logs and consider rollback"' ERR

# Argument parsing (omitted for brevity)
# ...

check_prerequisites() {
  [[ $EUID -eq 0 ]] || fail "Must run as root"
  command -v kubeadm >/dev/null || fail "kubeadm not found"
  command -v openssl >/dev/null || fail "openssl not found"
  command -v python3 >/dev/null || fail "python3 not found"
  command -v systemctl >/dev/null || fail "systemctl not found"
  [[ -f /etc/kubernetes/pki/ca.crt ]] || fail "Not a kubeadm control‑plane node"
  if [[ -e "${LOCK_FILE}" ]]; then
    fail "Lock file ${LOCK_FILE} exists, another task may be running"
  fi
  touch "${LOCK_FILE}"
}

# Detect expiring certs, backup, renew, restart, verify, etc.
# (Full implementation follows the original script logic.)

Features compared with common examples:

Lock file prevents concurrent executions.

Structured logging integrates with collection systems.

Supports --dry-run for rehearsal.

Creates etcd snapshots for extreme failure recovery.

Performs health checks to avoid a successful command but an unavailable control‑plane.

Go controller example

A custom CertificateRotation CRD drives per‑node renewal through a controller and a node‑agent DaemonSet. The controller ensures only one rotation runs per cluster, picks the next node, validates node health, creates a NodeRotationTask, and updates status.

type CertificateRotationReconciler struct { client.Client }

func (r *CertificateRotationReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // fetch CR, check phase, ensure single‑flight, pick next node, etc.
}

// ensureSingleFlight checks that no other Rotation with the same ClusterName is Running.
// pickNextNode returns the first node not listed in rotation.Status.CompletedNodes.
// ensureNodeHealthy verifies the NodeReady condition.
// createNodeTask creates a NodeRotationTask CR for the selected node.
// fail updates the Rotation status to Failed with a message.

The node‑agent watches its own NodeRotationTask and performs backup, optional etcd snapshot (for stacked etcd), runs kubeadm certs renew, restarts static Pods, validates health, and reports success or failure back to the controller.

Observability

Metrics

kubeadm_cert_expiry_days{cert_name,node,issuer,subject}

– remaining days before expiration.

kubeadm_cert_rotation_success_total{cluster,node}

kubeadm_cert_rotation_failure_total{cluster,node}

kubeadm_cert_rotation_duration_seconds{cluster,node}

kubeadm_cert_rotation_last_success_timestamp{cluster}

Logs

cluster, node, component, rotation_id, phase, result, duration_ms

Events

Start update, node drain, etcd snapshot success, renewal success, health check failure, rollback triggered, batch completed.

Prometheus exporter example

package main

import (
    "crypto/x509"
    "encoding/pem"
    "log"
    "net/http"
    "os"
    "path/filepath"
    "time"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var certExpiryDays = prometheus.NewGaugeVec(
    prometheus.GaugeOpts{Name: "kubeadm_cert_expiry_days", Help: "Remaining days before certificate expiration"},
    []string{"node", "cert_name", "issuer", "subject"},
)

func init() { prometheus.MustRegister(certExpiryDays) }

func loadCert(path string) (*x509.Certificate, error) { /* ... */ }
func scan(nodeName string) { /* walk /etc/kubernetes/pki/*.crt and set gauge */ }

func main() {
    node, _ := os.Hostname()
    go func() { ticker := time.NewTicker(5 * time.Minute); for { scan(node); <-ticker.C } }()
    http.Handle("/metrics", promhttp.Handler())
    log.Fatal(http.ListenAndServe(":9102", nil))
}

Corresponding alert rules detect certificates expiring within 30 days and already expired certificates.

Case study: 3‑node HA production cluster

Scenario: an e‑commerce core cluster with three control‑plane nodes, stacked etcd, 120+ workers, and a front‑end load balancer.

T‑30 days pre‑alert

Prometheus alerts when apiserver.crt, apiserver-etcd-client.crt, and front-proxy-client.crt have 29 days left.

The platform creates a change ticket and notifies the container platform team.

T‑7 days rehearsal

Run the full flow in a pre‑prod environment: backup, etcd snapshot, per‑node LB drain, renewal, static‑Pod restart, and regression verification. Focus on idempotency, alert accuracy, rollback script usability, and dashboard completeness.

T‑0 day production execution

Drain master-1 from the LB.

Execute the certificate script on the node.

Verify https://127.0.0.1:6443/healthz succeeds.

Check etcd member health.

Re‑add the node to the LB.

Observe for 10 minutes without anomalies before proceeding to the next node.

Repeat for master-2 and master-3.

Final acceptance checklist

kubectl get nodes

reports all nodes Ready. kubectl get pods -A shows no abnormal restarts.

No spike in apiserver 5xx responses.

etcd leader changes are within acceptable limits.

Core namespaces have no unexpected restarts.

Prometheus metrics show refreshed certificate expiry days.

Only after these checks is the renewal considered truly complete.

Common pitfalls and fixes

Pitfall 1: kubectl still fails after renewal

Root cause: admin.conf (and other kubeconfigs) still contain the old client certificate.

kubeadm init phase kubeconfig admin
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get nodes

Pitfall 2: apiserver does not start

Typical reasons:

Mismatch between apiserver-etcd-client.crt and the etcd CA.

apiserver restarts before etcd is healthy.

Manifest file was inadvertently modified.

Debug with:

crictl ps -a | grep kube-apiserver
journalctl -u kubelet -n 200 --no-pager
openssl x509 -in /etc/kubernetes/pki/apiserver-etcd-client.crt -noout -issuer -subject -dates

Pitfall 3: Only one control‑plane node was updated

Result: load balancer routes to a mix of updated and stale nodes, causing intermittent failures.

Fix: Ensure the rolling procedure is applied to every control‑plane node before declaring success.

Pitfall 4: Accidentally overwrote original certificates

Recovery steps:

Restore /etc/kubernetes/pki from backup.

Restore /etc/kubernetes/*.conf from backup.

If needed, restore the etcd snapshot.

Re‑launch static Pods.

Pitfall 5: External etcd cluster ignored

When using external etcd, treat Kubernetes control‑plane certificates and external etcd certificates as separate renewal streams; they may belong to different operational domains.

Rollback strategy

File‑level rollback

Use when certificates have been renewed but components have not yet stabilized. Restore /etc/kubernetes/pki, /etc/kubernetes/*.conf, static‑Pod manifests, and restart kubelet.

etcd data‑level rollback

Applicable when the renewal process caused etcd anomalies. Restore the pre‑change snapshot, re‑verify member health, then bring the apiserver back up.

Batch‑level circuit breaker

If a node fails during a rolling update, stop further node updates, keep already‑updated nodes unchanged, and trigger an manual incident response.

Best‑practice checklist

Governance layer : certificate inventory, 30‑day alerts, defined maintenance windows.

Tool layer : standardized script or platform task, dry‑run support, backup & rollback capability.

Architecture layer : HA clusters roll per node, stacked etcd includes peer certificates, external etcd handled separately.

Observability layer : expiry metrics, change‑event audit, failure alerts, health dashboards.

Organization layer : at least one rehearsal in a staging environment, runbook documented, clear change ownership.

Final recommendation

Do not treat certificate renewal as a one‑off command, a single incident, or a personal experience. It should be viewed as a lifecycle management mechanism for the control‑plane trust chain, a high‑availability governance capability, and a platform feature with audit, rollback, observability, and orchestration.

If your clusters are small, the production‑grade shell script is sufficient for the first phase. For multi‑cluster, multi‑business, HA environments, move beyond renew to a platform‑level certificate governance service.

Appendix A: Minimal command list

# 1. Check certificate expiration
kubeadm certs check-expiration

# 2. Backup
cp -R /etc/kubernetes /backup/kubernetes-$(date +%Y%m%d%H%M%S)

# 3. Renew all certificates
kubeadm certs renew all

# 4. Refresh kubeconfig files
kubeadm init phase kubeconfig admin
kubeadm init phase kubeconfig controller-manager
kubeadm init phase kubeconfig scheduler
kubeadm init phase kubeconfig kubelet --node-name "$(hostname)"

# 5. Restart kubelet (which reloads static Pods)
systemctl restart kubelet

# 6. Verify
KUBECONFIG=/etc/kubernetes/admin.conf kubectl get nodes
curl -k https://127.0.0.1:6443/healthz

Appendix B: Scope

This guide applies to clusters deployed with kubeadm where the control‑plane runs as static Pods. It is not intended for fully managed cloud services, heavily customized Kubernetes distributions, or environments that already use an enterprise‑grade certificate lifecycle platform.