Kubernetes Cluster Backup: Complete etcd & Config Protection with Scripts & Recovery Drills

Core Backup Principles

Kubernetes cluster backup is a critical operational practice that reduces recovery time from days to hours during incidents like accidental resource deletion, control plane crashes, or hardware failures. The strategy requires covering both dynamic data (etcd state: Deployments, Services, Pods, Secrets, ConfigMaps) and static configuration (certificates in /etc/kubernetes/pki, component manifests in /etc/kubernetes/manifests, kubelet config). Missing either leads to unrecoverable cluster state or inability to rebuild nodes.

Backup Strategy: Full + Incremental + Ad-hoc

Full backup : Daily at 02:00 (low traffic), retained 7 days — provides complete recovery baseline.

Incremental backup : Every 6 hours, retained 3 days — reduces resource consumption and backup window.

Ad-hoc backup : Before cluster upgrades or major changes, retained 15 days — enables quick rollback from operator error.

Storage Security: Offsite + Encryption + Least Privilege

Offsite storage : Backup files must leave cluster nodes; use object storage (OSS/S3) or dedicated backup server.

Encryption : etcd contains Secrets; encrypt with AES-256, manage keys via HashiCorp Vault.

Permissions : Backup directories/files restricted to root; object storage accessed via minimal IAM roles.

etcd Data Backup

Embedded etcd (kubeadm)

Critical prerequisite: etcdctl version must exactly match the etcd cluster version, otherwise backups corrupt. Steps:

Identify etcd pod and version:

ETCD_POD=$(kubectl get pods -n kube-system -l component=etcd --no-headers | awk '{print $1}')

and

ETCD_VERSION=$(kubectl exec -it ${ETCD_POD} -n kube-system -- etcdctl version | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+')

.

Fetch backup password from Vault (no hardcoding):

BACKUP_PASSWORD=$(vault kv get -field=password secret/k8s/backup)

.

Create backup directory with minimal permissions: mkdir -p /backup/etcd && chmod 700 /backup/etcd.

Execute snapshot inside the etcd pod using matching etcdctl:

kubectl exec -it ${ETCD_POD} -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt --key=/etc/kubernetes/pki/apiserver-etcd-client.key snapshot save /var/lib/etcd/snapshot-${TIMESTAMP}.db"

Copy snapshot out, compress, encrypt with OpenSSL using Vault password:

tar -zcvf ${BACKUP_FILE}.tar.gz ${BACKUP_FILE} && openssl enc -aes-256-cbc -salt -in ${BACKUP_FILE}.tar.gz -out ${BACKUP_FILE}.tar.gz.enc -k ${BACKUP_PASSWORD}

Upload to offsite OSS (AliCloud example, IAM role):

ossutil cp ${BACKUP_FILE}.tar.gz.enc oss://k8s-backup/etcd/$(date +%Y%m)/ --endpoint=oss-cn-beijing.aliyuncs.com

Clean local temp files.

External etcd Cluster

If local etcdctl version mismatches, download matching binary:

wget https://github.com/etcd-io/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz && tar zxvf ... && cp etcd-${ETCD_VERSION}-linux-amd64/etcdctl /usr/local/bin/ && chmod +x /usr/local/bin/etcdctl

Incremental backup uses WAL logs against a decrypted base snapshot:

openssl enc -d -aes-256-cbc -in oss://k8s-backup/etcd/202506/etcd-snapshot-20250601-020000.db.tar.gz.enc -out ${DECRYPTED_FILE}.tar.gz -k ${BACKUP_PASSWORD} && tar zxvf ${DECRYPTED_FILE}.tar.gz

Then:

ETCDCTL_API=3 etcdctl --endpoints=https://etcd-1:2379,https://etcd-2:2379,https://etcd-3:2379 --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/etcd-server.crt --key=/etc/etcd/pki/etcd-server.key snapshot save ${BACKUP_DIR}/etcd-snapshot-${TIMESTAMP}-inc.db --incremental=true --base-snapshot=${DECRYPTED_FILE}

Backup Validity Verification

Decrypt and check snapshot integrity:

openssl enc -d -aes-256-cbc -in ${BACKUP_FILE}.tar.gz.enc -out ${BACKUP_FILE}.tar.gz -k ${BACKUP_PASSWORD} && tar zxvf ${BACKUP_FILE}.tar.gz && ETCDCTL_API=3 etcdctl snapshot status ${BACKUP_FILE} --write-out=table

Acceptance criteria: TotalKey != 0, Corrupted = false, Version matches cluster.

Component Configuration Backup

Distinguish cluster-common configs (identical across nodes) from node-specific configs (unique per node, e.g., node IP, local storage) to avoid conflicts during multi-node restore.

Bash Script (Single/Multi-node)

Configurable via environment variables. Paths:

Common : /etc/kubernetes/pki/ca.crt, /etc/kubernetes/pki/ca.key, /etc/kubernetes/manifests/, /etc/cni/net.d/ Node-specific : /var/lib/kubelet/config.yaml, /etc/systemd/system/kubelet.service.d/, /etc/kubernetes/kubelet.conf, /var/lib/kubelet/pki/ Script creates timestamped tarball per node ( k8s-configs-${NODE_NAME}-${TIMESTAMP}.tar.gz), excludes *.old and etcd backup dir, verifies size > 1MB, alerts via WeCom webhook on failure, uploads to S3 with IAM role, retains local 7 days.

Ansible Multi-node Playbook

Targets host group k8s-nodes, copies script with mode 0700, sets VAULT_ADDR in /etc/profile, executes script, registers result, outputs per-host status, triggers alert on non-zero return code.

Automation & Monitoring

Systemd Timers (Replace Cron)

Service unit /etc/systemd/system/k8s-backup.service (Type=oneshot, User=root, depends on Vault, logs to /var/log/k8s-backup.log and error log). Timer unit /etc/systemd/system/k8s-backup.timer with OnCalendar=*-*-* 02:00:00, Persistent=true (catch up if offline), AccuracySec=1min. Enable:

systemctl daemon-reload && systemctl enable --now k8s-backup.timer

.

Prometheus + Grafana Monitoring

Core metrics: backup file existence (last 24h), file size deviation <30% from 7-day average, execution duration <10 minutes. PrometheusRule example:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: k8s-backup-alerts
  namespace: monitoring
spec:
  groups:
  - name: backup.rules
    rules:
    - alert: EtcdBackupMissing
      expr: |
        absent(file_exporter_filesize{path=~"/backup/etcd/etcd-snapshot-.*.db.enc"} > 0)
        or
        time() - max(file_exporter_filemtime{path=~"/backup/etcd/etcd-snapshot-.*.db.enc"}) > 86400
      for: 10m
      labels:
        severity: critical
      annotations:
        summary: "etcd backup missing or expired"
        description: "No valid etcd backup generated in last 24h; cluster unrecoverable on failure"
    - alert: EtcdBackupSizeAbnormal
      expr: |
        file_exporter_filesize{path=~"/backup/etcd/etcd-snapshot-.*.db.enc"} /
        avg_over_time(file_exporter_filesize{path=~"/backup/etcd/etcd-snapshot-.*.db.enc"}[7d])
        < 0.7 or > 1.3
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "etcd backup size abnormal"
        description: "Current backup size deviates >30% from 7-day average; possible data loss"

Production Recovery Drills

Control Plane Crash Recovery

Rebuild control plane node with original config.

Restore component configs from OSS:

aws s3 cp s3://k8s-backup/configs/control-plane-1/202506/k8s-configs-control-plane-1-20250601-020000.tar.gz /tmp/ && tar zxvf /tmp/... -C /

Restore etcd: download encrypted snapshot, decrypt with Vault password, extract, stop kubelet, wipe /var/lib/etcd/*, run etcdctl snapshot restore with

--data-dir=/var/lib/etcd --name=control-plane-1 --initial-cluster=control-plane-1=https://192.168.1.100:2380 --initial-cluster-token=etcd-cluster-token --initial-advertise-peer-urls=https://192.168.1.100:2380

.

Restart kubelet, wait for API server ready:

kubectl wait --for=condition=Ready pod -n kube-system -l component=kube-apiserver --timeout=300s

.

Verify: compare pod count pre/post, test core service connectivity (

kubectl exec -it -n default nginx-pod -- curl -s http://api-service

).

Drill Cadence & Documentation

Quarterly full recovery drill, monthly incremental restore validation.

Maintain K8s Cluster Recovery Runbook with: backup locations & decryption steps, per-scenario flowcharts, verification checklist (resource counts, service availability, data consistency).

Summary

The article's core improvements address prior pain points: (1) Secrets management via Vault eliminates hardcoded passwords; (2) Strict etcdctl /etcd version matching with concrete download steps; (3) Practical WAL-based incremental backup implementation; (4) Offsite upload commands using IAM roles (no static keys); (5) End-to-end control plane crash recovery procedure. Backup is not a one-time task — regular validation, script updates, and documentation sync are essential. Implementing this framework yields a reliable K8s backup system that ensures business continuity.