Mastering etcd: The Core of Kubernetes State Management and High‑Availability

What is etcd?

etcd is a distributed, highly‑available, strongly consistent key‑value store developed by the CoreOS team and now a CNCF graduated project alongside Kubernetes and Prometheus.

Core features:

Strong consistency : based on the Raft algorithm, guaranteeing data consistency across the cluster.

High availability : tolerates node failures without service interruption.

Key‑value storage : organizes data as hierarchical keys and supports watch mechanisms.

In Kubernetes, etcd is the sole state storage component; all other components are stateless and interact with etcd through the kube‑apiserver.

What does etcd store for Kubernetes?

etcd acts as the Kubernetes database, persisting definitions and status of all objects, including:

Nodes – status, capacity, etc.

Pods – specifications, status, scheduling info, IP addresses.

Services and Endpoints – service abstraction and dynamic pod endpoints.

ConfigMaps and Secrets – configuration data and sensitive information.

Controller objects – ReplicaSet, Deployment, StatefulSet, etc.

RBAC rules – roles and bindings.

Network and storage – NetworkPolicy, PersistentVolumeClaim, and related resources.

One‑sentence summary : Every piece of Kubernetes cluster state ultimately resides in etcd.

Interaction and workflow

etcd does not communicate directly with other components; all traffic passes through the kube‑apiserver.

User request: a user submits an operation via kubectl or the API.

apiserver validation: authentication, authorization, and admission control.

Write to etcd: the apiserver persists the desired state in etcd.

Controller notification: controllers watch for resource changes.

Scheduler decision: the scheduler sees a new Pod, selects a node, and writes the assignment back to etcd.

Kubelet execution: kubelet on the chosen node observes the schedule result and starts the Pod.

Status update: kubelet reports the Pod’s actual status back to etcd.

This declarative API plus watch mechanism forms the control loop that continuously drives the actual state toward the desired state.

Key mechanisms of etcd

Watch mechanism : pushes changes to listeners in real time, avoiding polling.

Lease and heartbeat : nodes renew leases; expired leases mark nodes as unhealthy.

Transaction support : multi‑key atomic operations ensure complex updates are consistent.

Revision numbers : each write increments a global revision used for watches and optimistic locking.

Data model

etcd stores data in a hierarchical key space similar to a filesystem.

/registry/pods/default/nginx

/registry/secrets/default/db-credentials

Additional concepts:

Revision : a globally monotonic version number.

Optimistic lock (ModRevision) : uses the revision to prevent concurrent write conflicts.

High‑availability deployment

Cluster size: typically 3, 5, or 7 nodes (odd numbers simplify leader election).

Raft algorithm: elects a leader that handles all write requests.

Deployment modes:

Static Pod: when using kubeadm, etcd runs as a static pod on control‑plane nodes.

Standalone cluster: production environments usually deploy etcd as an independent cluster to reduce interference.

Operations and best practices

Backup : regularly take snapshots, e.g.

etcdctl snapshot save backup.db

etcdctl snapshot restore backup.db

Monitoring metrics : request latency (read/write), leader change count, database size (default 2 GB limit), node health.

Performance tuning : use SSDs, place the etcd data directory on a dedicated disk.

Failure and recovery

Single‑node failure: Raft automatically recovers.

Majority failure: the cluster becomes unavailable for writes and requires manual intervention.

Disaster recovery steps:

Restore a new node from a snapshot.

Replace failed nodes and update the cluster configuration.

Best practice: regularly rehearse etcd recovery in production.

Security

TLS encryption: all traffic must use TLS (provide --cert-file, --key-file, --trusted-ca-file).

Least‑privilege access: only the kube‑apiserver should be allowed to talk to etcd.

Firewall and RBAC: further restrict access sources.

Performance and capacity planning

Database size: keep below 2 GB–8 GB for optimal performance.

Node count: 3–5 nodes suit most production workloads.

Large clusters: deploy etcd independently and optimize disk and network resources.

One‑sentence conclusion : The stability and performance of etcd directly determine the overall reliability and throughput of a Kubernetes cluster.