Kubernetes etcd Operations Guide: From Backup & Restore to Cluster Performance Tuning

Problem Background

In production, etcd failures often have simple root causes but high impact, such as missing snapshots before a kubeadm upgrade, disk full errors, I/O latency spikes, or accidental use of the wrong --data-dir during etcdctl snapshot restore. Since etcd stores the entire state of a Kubernetes cluster, any outage can bring the whole cluster down.

etcd Role in Kubernetes

All cluster objects—Pods, Services, ConfigMaps, Secrets, CRDs, RBAC resources, etc.—are stored as flat keys in etcd (e.g., /registry/pods/default/nginx-7c8d9f-x7k2m). The API server is the only client that reads and writes these keys.

Version Compatibility

Kubernetes 1.24‑1.27 uses etcd 3.5.4‑3.5.6

Kubernetes 1.28‑1.29 uses etcd 3.5.9‑3.5.12

Kubernetes 1.30+ prefers etcd 3.5.13+

Do not skip major etcd versions; a data migration is required when moving from 3.4 to 3.5.

Manual Backup

# Connect to a control‑plane node (kubeadm example)
ssh control-plane-1

# Verify the etcd pod
sudo crictl ps | grep etcd

# Identify the data directory (usually /var/lib/etcd)
sudo crictl inspect <etcd-container-id> | grep -i 'workdir\|datadir'

# List etcd certificates
ls /etc/kubernetes/pki/etcd/

# Create a snapshot
sudo ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify the snapshot
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-20260605-103045.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Typical output shows totalKey in the thousands and totalSize between 100 MiB and 500 MiB. Run backups during low‑traffic windows to avoid I/O spikes.

Automated Backup with CronJob

# /opt/scripts/etcd-backup.sh
#!/bin/bash
set -euo pipefail
BACKUP_DIR="/backup/etcd"
RETENTION_DAYS=14
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_FILE="$BACKUP_DIR/etcd-$TIMESTAMP.db"
mkdir -p "$BACKUP_DIR"
export ETCDCTL_API=3
etcdctl snapshot save "$BACKUP_FILE" \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
etcdctl snapshot status "$BACKUP_FILE" \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
# Upload to remote storage (e.g., NFS, S3, OSS)
cp "$BACKUP_FILE" /nfs/etcd-backup/
# Clean up old local backups
find "$BACKUP_DIR" -name 'etcd-*.db' -mtime +7 -delete
# Clean up old remote backups
find /nfs/etcd-backup/ -name 'etcd-*.db' -mtime +$RETENTION_DAYS -delete
echo "[$(date)] etcd backup ok: $BACKUP_FILE"

# Add to crontab (run at 03:30 daily, staggered by 30 minutes)
30 3 * * * /opt/scripts/etcd-backup.sh >> /var/log/etcd-backup.log 2>&1

Alternatively, use the etcd-backup-restore-operator or a native CronJob that mounts the certificates and runs etcdctl snapshot save inside a container.

Backup Strategy Summary

Frequency: at least daily; for critical workloads, every 4‑6 hours.

Retention: keep local backups for 7 days, remote copies for ≥30 days.

Verification: perform a restore test at least once per month.

Encryption: use SSE‑KMS/SSE‑S3 for cloud storage and encrypt files locally with gpg or openssl.

Recovery Scenarios

Single Node Failure

With a 3‑node cluster, losing one node still provides a quorum (2). Re‑add the node after fixing the underlying issue and run etcdctl member remove <member-id> if necessary, then regenerate the static pod manifest with kubeadm init phase etcd local.

Full Cluster Failure

# Stop all control‑plane components
sudo systemctl stop kube-apiserver
sudo mv /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/manifests/etcd.yaml.bak
# Remove old data
sudo mv /var/lib/etcd /var/lib/etcd.bak.$(date +%s)
# Restore from snapshot
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-20260605-103045.db \
  --name=control-plane-1 \
  --initial-cluster=control-plane-1=https://10.0.0.1:2380,control-plane-2=https://10.0.0.2:2380,control-plane-3=https://10.0.0.3:2380 \
  --initial-advertise-peer-urls=https://10.0.0.1:2380 \
  --data-dir=/var/lib/etcd
# Restore the static pod manifest and start the pod
sudo mv /etc/kubernetes/manifests/etcd.yaml.bak /etc/kubernetes/manifests/etcd.yaml
sudo crictl ps -a | grep etcd   # wait for pod to become Ready
sudo systemctl start kube-apiserver

CRD Deletion

# Retrieve the CRD definition from the snapshot
ETCDCTL_API=3 etcdctl get /registry/apiextensions.k8s.io/customresourcedefinitions/<name> \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key -o json | jq -r '.value' | base64 -d > crd.yaml
# Re‑apply the CRD
kubectl apply -f crd.yaml

Performance Tuning

Key tuning dimensions are disk, memory, CPU, network, and etcd flags.

Startup Flags (excerpt from /etc/kubernetes/manifests/etcd.yaml )

spec:
  containers:
  - command:
    - etcd
    - --name=control-plane-1
    - --data-dir=/var/lib/etcd
    - --listen-client-urls=https://10.0.0.1:2379
    - --advertise-client-urls=https://10.0.0.1:2379
    - --listen-peer-urls=https://10.0.0.1:2380
    - --initial-advertise-peer-urls=https://10.0.0.1:2380
    - --initial-cluster=control-plane-1=https://10.0.0.1:2380,...
    - --quota-backend-bytes=8589934592   # 8 GB hard limit
    - --auto-compaction-mode=periodic
    - --auto-compaction-retention=8h
    - --max-snapshots=5
    - --max-wals=5
    - --election-timeout=5000        # ms, increase to avoid frequent elections
    - --heartbeat-interval=500       # ms, increase to match election timeout
    - --snapshot-count=10000          # trigger snapshots more often in write‑heavy workloads
    - --listen-metrics-urls=http://0.0.0.0:2381

Important flags: --quota-backend-bytes: hard DB size limit; exceeding triggers mvcc: database space exceeded. Production clusters typically set 8 GB‑32 GB. --auto-compaction-mode and --auto-compaction-retention: periodic MVCC compaction to free old revisions. --election-timeout and --heartbeat-interval: longer values reduce election churn caused by network jitter or disk I/O spikes.

Disk I/O

etcd relies heavily on low‑latency SSD/NVMe. Monitor wal_fsync_duration_seconds P99; keep it <10 ms (or <25 ms for write‑intensive clusters). Use iostat -dx 1 to watch %util, await, and svctm. Prefer the none or deadline I/O scheduler; avoid cfq.

Memory & CPU

etcd process uses 1‑2 GB; allocate ample page cache. A minimum of 8 GB RAM on the control‑plane node is recommended. CPU should be at least 4 cores; 8 cores give headroom for write bursts.

Network

Peer RTT should stay below 10 ms; keep etcd nodes in the same rack or availability zone to avoid cross‑zone latency.

Monitoring

etcd exposes metrics on port 2381 (add --listen-metrics-urls=http://0.0.0.0:2381 if not already). Key Prometheus metrics and recommended alert thresholds: etcd_server_leader_changes_seen_total: alert if >2 changes in 1 minute. etcd_disk_writes_total: warn if sustained >2000 writes/s. etcd_disk_wal_fsync_duration_seconds (P99): warn if >10 ms. etcd_disk_backend_commit_duration_seconds (P99): warn if >25 ms. etcd_debugging_mvcc_db_total_size_in_bytes: warn if >80 % of --quota-backend-bytes.

Import Grafana dashboard ID 3070 (community etcd dashboard) and customize panels for topology, latency, traffic, capacity, and health.

Common Failure Troubleshooting

apiserver connection refused

# Check apiserver logs
sudo journalctl -u kube-apiserver -n 200
# Verify etcd health
ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Typical causes: etcd not started, certificate mismatch, firewall blocking port 2379.

"mvcc: database space exceeded"

# Inspect DB size
ETCDCTL_API=3 etcdctl endpoint status -w table \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
# Temporarily increase quota (edit etcd.yaml)
# --quota-backend-bytes=17179869184   # 16 GB
# Run defragmentation
ETCDCTL_API=3 etcdctl defrag \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Frequent leader elections

# Check network latency between peers
ping <peer-ip>
# Examine fsync latency
curl -s http://127.0.0.1:2381/metrics | grep wal_fsync_duration_seconds
# Review etcd logs for election messages
sudo crictl logs <etcd-container-id> | grep -i 'election\|leader\|heartbeat'
# Tune timeouts
# Increase --election-timeout to 5000 ms and --heartbeat-interval to 500 ms in etcd.yaml

Snapshot restore failure

# Verify snapshot integrity
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-20260605.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
# Try restoring with a different etcd version in a container
docker run --rm -v /backup:/backup quay.io/coreos/etcd:v3.5.13 \
  etcdctl snapshot status /backup/etcd-20260605.db

Root cause is usually an in‑flight write during snapshot creation or corrupted upload.

Kubernetes upgrade breaks etcd

# Review kubeadm upgrade plan
sudo kubeadm upgrade plan
# Upgrade kubeadm, then apply the upgrade (etcd image is updated automatically)
sudo apt-get install -y kubeadm=1.30.x-00
sudo kubeadm upgrade apply v1.30.x
# If etcd fails to start, check the manifest for unsupported flags or expired certificates.

etcd Upgrade Procedure

# 1. Take a snapshot (mandatory)
sudo /opt/scripts/etcd-backup.sh
# 2. Upgrade kubeadm package
sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.30.x-00 && sudo apt-mark hold kubeadm
# 3. Review the upgrade plan
sudo kubeadm upgrade plan
# 4. Apply the upgrade (this pulls a newer etcd image)
sudo kubeadm upgrade apply v1.30.x
# 5. Upgrade remaining control‑plane nodes
sudo kubeadm upgrade node
# 6. Upgrade kubelet and kubectl
sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.30.x-00 kubectl=1.30.x-00 && sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet

Etcd data format migration (3.4 → 3.5) happens automatically. Verify cluster health after each node upgrade.

etcd Encryption at Rest

# /etc/kubernetes/etcd-encryption.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64‑encoded‑32‑byte‑key>
  - identity: {}

Add the flag

--encryption-provider-config=/etc/kubernetes/etcd-encryption.yaml

to the API server. Rotate keys by placing the new key first, restarting the API server, letting secrets re‑encrypt, then removing the old key.

FAQ Highlights

Memory: etcd process uses 1‑2 GB; allocate at least 8 GB RAM on control‑plane nodes.

SSD requirement: SSD/NVMe is mandatory; mechanical disks cause high fsync latency.

Quorum: a 3‑node cluster tolerates one node loss.

Snapshot restore loses data after the snapshot point; frequent snapshots reduce loss.

Do not store etcd data on NFS or NAS; use local SSD.

Encryption keys must be stored securely (KMS, Vault); loss makes secrets unrecoverable.

Conclusion

etcd is the brain of Kubernetes. By implementing daily backups, regular restore drills, comprehensive monitoring with alert thresholds, proper resource sizing (disk, memory, CPU), and cautious upgrade practices, operators can keep the cluster stable and avoid catastrophic outages.