Rescue Expired Kubernetes Certificates Offline: A 4‑Step Emergency Guide

Introduction

Kubernetes clusters deployed in highly regulated environments (finance, government, defense) often run in isolated internal networks where only internally‑issued X.509 certificates are permitted. Expiration of these certificates can cause sudden loss of API server connectivity and control‑plane failure.

Key Challenges

Network isolation: No external connectivity; all operations must be performed offline.

Compliance: Only certificates signed by an internal CA are allowed; self‑signed or public CAs are prohibited.

Availability SLA: Critical services require 99.99% uptime, with downtime limited to less than five minutes.

Emergency Rescue: Four‑Step Offline Certificate Issuance

Step 1 – Build Offline Rescue Toolkit

# Toolkit directory layout
cert-rescue-kit/
├── bin/
│   ├── cfssl_1.6.4_linux_amd64   # cfssl signing binary
│   ├── cfssljson_1.6.4_linux_amd64
│   └── k8s-cert-checker          # helper script to validate existing certs
├── conf/
│   ├── ca-config.json            # CA profile and expiry settings
│   ├── ca-csr.json               # root CA CSR template
│   └── apiserver-csr.json        # API server CSR template
└── scripts/
    ├── backup-certs.sh           # backs up /etc/kubernetes/pki
    └── deploy-certs.sh           # copies new certs into place

Step 2 – Generate Root CA (run once)

# Create CA private key and self‑signed certificate
./cfssl gencert -initca conf/ca-csr.json | ./cfssljson -bare ca
# Resulting files:
# ca.pem      – CA certificate
# ca-key.pem  – CA private key (store securely)

Step 3 – Issue Component Certificates

# Generate API server private key
openssl genrsa -out apiserver.key 2048

# Generate CSR and sign with the root CA
./cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=conf/ca-config.json \
  -hostname=10.0.0.1,kubernetes.default.svc,kubernetes.default,localhost,127.0.0.1 \
  -profile=kubernetes \
  conf/apiserver-csr.json | ./cfssljson -bare apiserver

# Important flags:
# -hostname must list every IP/DNS the API server will be reachable at.
# -profile must match the "kubernetes" profile defined in ca-config.json.

Step 4 – Hot‑Replace Cluster Certificates

# 1. Backup current PKI directory
./backup-certs.sh /etc/kubernetes/pki

# 2. Deploy the newly generated certs
cp apiserver.pem /etc/kubernetes/pki/
cp apiserver-key.pem /etc/kubernetes/pki/

# 3. Rolling restart of control‑plane components (order matters)
systemctl restart kube-apiserver kube-controller-manager kube-scheduler

# 4. Update kubeconfig to point to the new client certificate
sed -i 's|client-certificate:.*|client-certificate: /etc/kubernetes/pki/apiserver.pem|' /etc/kubernetes/admin.conf

Risk note: Restart components in the sequence API Server → Controller Manager → Scheduler. In production, perform a rolling restart node‑by‑node to avoid service interruption.

Long‑Term Automation: Certificate Lifecycle Management

Deploy HashiCorp Vault as the internal CA and use Cert‑Manager to request and rotate certificates. Prometheus monitors expiry and triggers alerts.

# ClusterIssuer that talks to Vault
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
spec:
  vault:
    path: pki/sign/k8s-cluster
    server: https://vault.example.com
    caBundle: LS0tLS1CRUdJ...   # Base64‑encoded CA bundle
---
# Certificate resource for the API server
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: apiserver-cert
spec:
  secretName: apiserver-tls
  duration: 2160h   # 90 days
  renewBefore: 360h # renew 15 days before expiry
  issuerRef:
    name: vault-issuer
    kind: ClusterIssuer
  dnsNames:
    - kubernetes.default.svc.cluster.local
    - k8s-api.example.com

Prometheus alert rule to detect certificates expiring within 30 days:

# Alert when remaining validity < 30 days
- alert: K8sCertificateExpiry
  expr: kubelet_server_certificate_expiration_seconds{job="kubelet"}/86400 < 30
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "Certificate {{ $labels.host }} expires in <30 days"
    description: "Path: {{ $labels.path }}"

Compliance Audit & Disaster Recovery Design

Key audit items include strict file permissions (0400) for private keys, secure storage of the CA key in Vault, and an immutable audit log of all certificate operations.

# Example audit table schema
CREATE TABLE cert_audit (
  id INT PRIMARY KEY,
  cert_name VARCHAR(255),
  action_type ENUM('CREATE','UPDATE','REVOKE'),
  expire_date DATETIME,
  operator VARCHAR(64),
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Disaster‑recovery measures:

Regular offline backup of the /etc/kubernetes/pki directory.

Multi‑CA trust model using a ConfigMap that bundles a primary and a backup CA certificate.

# ConfigMap containing both CA bundles
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-apiserver-ca
  namespace: kube-system
data:
  ca-bundle.crt: |
    -----BEGIN CERTIFICATE-----
    # Primary CA certificate
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    # Backup CA certificate
    -----END CERTIFICATE-----

Real‑World Case: Banking Private‑Cloud Certificate Incident

Background: 200‑node cluster running >300 micro‑services experienced an API server certificate expiry, causing the scheduler to lose contact.

Emergency actions: Used the offline rescue toolkit to generate and hot‑replace the API server certificate in ~15 minutes , then performed a rolling restart of control‑plane components via Ansible.

Root cause: Manual certificate management without monitoring and a one‑year validity period.

Improvements: Deployed Cert‑Manager for fully automated rotation and introduced a dual‑CA trust architecture.

Results: Mean‑time‑to‑repair dropped from four hours to five minutes; no certificate‑related outages occurred for an entire year.

Resources

HashiCorp Vault K8s guide: https://developer.hashicorp.com/vault/docs/platform/k8s

Cert‑Manager documentation: https://cert-manager.io/docs/

K8s certificate management whitepaper: https://github.com/k8s-cert/cert-whitepaper

Getting the Offline Toolkit

# Clone the rescue‑kit repository and initialise
git clone https://example.com/k8s-cert/rescue-kit.git
cd rescue-kit && ./init.sh