Destruction and Rebirth: Deep Dive into ETCD Backup and Restore for Kubernetes Clusters

1. A Real Incident that Triggered the Deep Dive

At 02:00 a retail platform’s pre‑sale environment experienced widespread timeouts, pending pods, CoreDNS endpoint failures, HPA stalls, and secret lookup errors. The team initially blamed apiserver jitter, but discovered the root cause: a high‑risk operation that replaced an ETCD node’s member/snap/db after an accidental deletion of critical resources, leaving the node’s member ID, WAL and snapshot metadata out of sync, breaking the Raft quorum and rendering the control plane unavailable.

Using a snapshot taken 20 minutes earlier, the cluster was rebuilt and services recovered within 35 minutes.

2. Why ETCD Is the "Brain" of Kubernetes

All critical control‑plane state—namespaces, pods, deployments, configmaps, secrets, RBAC, CRDs, node status, etc.—is stored in ETCD. When ETCD fails, controllers lose the current version of resources, the scheduler cannot see up‑to‑date nodes, the apiserver cannot reliably persist changes, and new requests may be dropped or write incomplete state.

3. How ETCD Works

3.1 Three Core Layers: Raft, WAL, BoltDB

ETCD consists of:

Raft consensus layer

WAL (write‑ahead log) persistence layer

BoltDB state‑machine storage layer

The write path is: client → etcd leader → Raft proposal → majority ACK → append to WAL → apply to MVCC store → persist in BoltDB → response.

3.2 MVCC and Revision

Each write increments a global revision. Kubernetes objects use resourceVersion (derived from this revision) for watches and controller change detection. Restoring from a snapshot can only bring the state back to the revision captured; any later changes are lost unless an incremental log is kept.

3.3 Compaction and Defragmentation

Compaction removes old revisions to bound history size, while defragmentation reclaims disk space in BoltDB. Both affect backup size and recovery time, so snapshots must be taken before compaction windows and stored compressed.

4. Why Directly Copying /var/lib/etcd Is Wrong

Consistency risk: copying while WAL is being written can capture a partially written state.

Identity pollution: the directory contains member IDs and cluster metadata; copying it to another node can cause member‑ID conflicts.

Semantic error: ETCD restore should create a **new** Raft cluster from a logical snapshot, not overwrite an existing node’s data directory.

5. Correct Backup Method – Snapshot API

Use the official snapshot commands:

ETCDCTL_API=3 etcdctl snapshot save snapshot.db
ETCDCTL_API=3 etcdctl snapshot status snapshot.db -w table

Key points:

Snapshot is a consistent view of the entire keyspace.

Never copy files directly.

Validate the snapshot with snapshot status before proceeding.

6. Defining the Goal – Backup vs. Restore

RPO (maximum data loss) and RTO (time to recover) drive the design. For a high‑traffic e‑commerce platform, a 1‑hour snapshot interval may be acceptable for dev/test, but production requires tighter windows.

7. Production‑Grade Backup Architecture

A recommended architecture includes: CronJob to trigger snapshots on a control‑plane node.

A dedicated backup worker (written in Go) handling TLS, timeouts, compression, encryption, and upload.

Object storage (S3/MinIO) for durable, versioned storage.

Prometheus metrics and Alertmanager for observability.

Cross‑region replication for disaster‑recovery.

7.1 CronJob Details

apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: kube-system
spec:
  schedule: "*/15 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: etcd-backup
          restartPolicy: OnFailure
          containers:
          - name: backup-worker
            image: registry.example.com/platform/etcd-backup:v1.2.0
            args:
            - "--cluster-name=prod-trade"
            - "--endpoints=https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379"
            - "--s3-endpoint=https://minio.infra.local"
            - "--s3-bucket=platform-etcd-backup"
            - "--compression=zstd"
            - "--snapshot-timeout=8m"
            - "--upload-timeout=10m"
            - "--retention=336h"
            env:
            - name: S3_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: etcd-backup-s3
                  key: access-key
            - name: S3_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: etcd-backup-s3
                  key: secret-key
            - name: ENCRYPTION_KEY
              valueFrom:
                secretKeyRef:
                  name: etcd-backup-encryption
                  key: aes256-key

7.2 Go Backup Program Highlights

Parse endpoints into a slice.

Create an etcd client with TLS.

Save snapshot, verify size > 0, compute SHA‑256.

Compress & encrypt (AES‑GCM, Base64‑encoded 32‑byte key).

Upload to object storage with retries.

Emit structured Prometheus metrics (success, failure, duration, size, revision).

// Simplified snippet
func RunBackup(ctx context.Context, cfg Config, logger *slog.Logger) error {
    cli, err := newEtcdClient(cfg)
    if err != nil { return fmt.Errorf("create etcd client: %w", err) }
    defer cli.Close()
    // ... snapshot, verify, encrypt, upload ...
    logger.Info("backup completed", "cluster", cfg.ClusterName, "revision", metadata.Revision)
    return nil
}

8. Verification Layers – From File to Restore

Three verification steps are recommended:

File‑level: non‑zero size, correct SHA‑256, matching ETag after upload.

Snapshot‑level: etcdctl snapshot status fields – revision, total keys, total size, hash.

Restore‑level: periodic “disaster‑recovery drill” that decrypts the snapshot, restores to a fresh node, starts a temporary etcd, and runs etcdctl get "" --prefix --keys-only or custom validation scripts.

9. Full‑Cluster Restore Procedure (3‑Node Example)

Confirm no healthy majority and select the baseline snapshot.

Stop static etcd and apiserver pods ( mv /etc/kubernetes/manifests/etcd.yaml /tmp/etcd.yaml.bak).

Backup the broken /var/lib/etcd directory for forensics.

Decrypt the snapshot if encrypted.

Run etcdctl snapshot restore on each node with unique --name and matching --initial-cluster but different --initial-advertise-peer-urls.

Restore static pod manifests, let the new etcd form a healthy cluster, then bring back apiserver.

Validate health ( etcdctl endpoint health), member list, and core Kubernetes resources (nodes, namespaces, pods, endpoints).

10. Post‑Restore Challenges

apiserver cache may hold stale resourceVersion causing "object has been modified" errors – restart all apiserver instances.

Running pods may temporarily diverge from the restored state, leading to duplicate creations or orphan clean‑ups.

Operators, webhooks, and custom controllers can generate a storm of watch re‑establishments; stagger their restart or add back‑off.

11. Scaling Considerations for Large Clusters

Snapshot size grows to GBs, increasing I/O, network, and memory usage during restore.

Compaction windows must be larger than the snapshot interval to avoid losing events.

High‑frequency changes (leases, short‑lived jobs, large ConfigMaps) amplify ETCD pressure; prune unnecessary objects and perform regular compaction/defragmentation.

12. Incremental Event Archiving (Optional RPO Boost)

Combine periodic full snapshots with a watch‑based event log (Kafka or object storage). On restore, replay events after applying the latest snapshot. This reduces RPO to minutes or seconds but adds complexity: reliable event storage, handling compaction windows, and idempotent replay.

13. Observability & Alerting

Export metrics such as etcd_backup_success_total, etcd_backup_failure_total, etcd_backup_duration_seconds, etcd_backup_snapshot_size_bytes, and ETCD internal metrics (fsync latency, leader changes, backend size). Alert on consecutive failures, stale successful timestamps, abnormal size growth, or revision stalls.

14. Security Practices

Transport encryption (TLS) for snapshot upload.

At‑rest encryption of backup files (AES‑GCM with keys stored in a KMS/Vault).

Object‑storage versioning or Object Lock.

Least‑privilege IAM for backup agents.

Audit logs for every backup and restore operation.

15. Common Pitfalls

Storing backups only on local disks – they disappear with the node.

Assuming a successful snapshot guarantees restore – without drills you may miss certificate paths, version mismatches, or command errors.

Copying a node’s data-dir directly – leads to member‑ID conflicts.

Over‑frequent snapshots – excessive I/O and storage consumption.

Restoring ETCD without re‑creating static control‑plane pods – apiserver will stay down.

16. Practical Three‑Stage Adoption Plan

Foundation : Enable etcdctl snapshot save every 15‑30 min, encrypt, upload to object storage, generate metadata, and set up failure alerts.

Recovery : Write a runbook, automate restore scripts, conduct monthly disaster‑recovery drills, and verify core resources after each restore.

Advanced RPO : Evaluate incremental watch archiving, ETCD Learner replicas for cross‑region DR, and tighter GitOps integration for post‑restore compensation.

By following this roadmap, teams move from “we can take a snapshot” to “we can reliably bring the entire control plane back online under pressure.”