Mastering Kubernetes StatefulSets: Persistent Storage, Stable IDs, and Ordered Scaling

StatefulSet is a Kubernetes controller designed to manage stateful applications that require stable network identities, persistent storage, and ordered pod lifecycle management, unlike Deployments which target stateless services.

Key Features of StatefulSet

Stable persistent storage using PersistentVolumeClaims (PVC) so that a pod retains its data after rescheduling.

Stable network identity via a Headless Service, ensuring the pod name and hostname remain unchanged.

Ordered deployment and scaling: pods are created and become ready sequentially (0 to N‑1) using init containers.

Ordered termination: pods are terminated in reverse order (N‑1 to 0).

Core Components

Headless Service that generates DNS records for each pod.

volumeClaimTemplates that create PVCs bound to PVs for each pod.

The StatefulSet object itself, which defines the pod template.

Typical YAML Definition

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

The example creates a headless Service named nginx and a StatefulSet web with two replicas, each mounting its own PVC.

Deploying and Observing Pods

kubectl apply -f statefulset.yaml

Use two terminal windows: one runs kubectl get pods -w -l app=nginx to watch pod creation, the other applies the manifest. The output shows the service and StatefulSet creation, followed by pod statuses progressing from Pending to ContainerCreating and finally Running.

Ordered Pod Creation

Pods are created in ordinal order (e.g., web-0 then web-1). The second pod starts only after the first reaches Running and Ready state.

Stable Network Identity

Each pod receives a stable hostname derived from <statefulset-name>-<ordinal>. Verify with:

for i in 0 1; do kubectl exec web-$i -- sh -c 'hostname'; done

To resolve DNS entries, install dnsutils in a pod and run nslookup against the fully qualified domain name, e.g., web-0.nginx.default.svc.cluster.local.

Deleting and Re‑creating Pods

kubectl delete pod -l app=nginx

The StatefulSet controller automatically recreates the pods, preserving their stable identities and PVC bindings.

Inspecting Persistent Volumes

kubectl get pvc -l app=nginx

Each pod has its own PVC (e.g., www-web-0, www-web-1) bound to a PV, ensuring data isolation.

Writing Hostname to Web Content

for i in 0 1 2; do kubectl exec web-$i -- sh -c 'echo $(hostname) > /usr/share/nginx/html/index.html'; done

After updating the index files, verify the content with:

for i in 0 1 2; do kubectl exec web-$i -- curl localhost; done

If a 403 Forbidden response appears, adjust the directory permissions:

for i in 0 1; do kubectl exec web-$i -- chmod 755 /usr/share/nginx/html; done

Key Takeaways

StatefulSet provides ordered, reliable deployment of stateful workloads.

Headless Services and PVCs give each pod a stable DNS name and dedicated storage.

Pod lifecycle events (creation, deletion, scaling) follow strict ordering, which is observable via kubectl commands.

Understanding these concepts is essential for running databases, message queues, and other stateful services on Kubernetes.