Mastering Kubernetes Persistent Storage: Volumes, PVs, PVCs & StorageClasses

Introduction

Kubernetes separates transient compute from persistent data. Pods can be recreated or moved, so data loss is unacceptable. Kubernetes therefore provides a decoupled, layered storage model where applications request storage through PersistentVolumeClaims (PVCs) instead of accessing disks directly.

Overall Storage Object Relationships

The storage hierarchy consists of four layers:

Pod : defines how a volume is used.

PersistentVolumeClaim (PVC) : declares the size and capabilities required by the application.

PersistentVolume (PV) : represents an actual storage resource provisioned by the cluster.

StorageClass : describes how a PV should be provisioned (provisioner, parameters, policies).

Applications interact only with PVCs, remaining agnostic to the underlying implementation.

1. Volume – Pod‑Level Storage Abstraction

Volumes are declared in a Pod spec and can be of many types. Example:

apiVersion: v1
kind: Pod
metadata:
  name: volume-demo
spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: html-volume
      mountPath: /usr/share/nginx/html
    - name: config-volume
      mountPath: /etc/nginx/conf.d
  volumes:
  - name: html-volume
    emptyDir: {}
  - name: config-volume
    configMap:
      name: nginx-config

Common volume types:

emptyDir : temporary storage that lives as long as the Pod.

hostPath : node‑local path, useful only for debugging.

configMap / secret : store configuration data or certificates.

downwardAPI : expose Pod metadata as files.

persistentVolumeClaim : reference a PVC for production‑grade persistent data.

2. PersistentVolume (PV) – Real Storage Resource

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-demo
  labels:
    environment: production
spec:
  capacity:
    storage: 100Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-storage
  nfs:
    server: 192.168.1.100
    path: /data/nfs-share

Key PV parameters:

capacity : total storage size.

accessModes : RWO, RWX, ROX, etc.

volumeMode : Filesystem or Block.

persistentVolumeReclaimPolicy : Retain, Delete, or Recycle (deprecated).

storageClassName : links the PV to a StorageClass.

3. PersistentVolumeClaim (PVC) – Storage Demand Specification

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast
  resources:
    requests:
      storage: 50Gi

PVC is not the storage itself; it is a “storage requirement document”.

Kubernetes matches a PVC to an existing PV that satisfies the request, or it creates a new PV dynamically via a StorageClass.

4. StorageClass – Dynamic Provisioning Engine

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

StorageClass decouples applications from the concrete storage implementation and defines default performance, encryption, and expansion policies. It is the entry point for platform‑level storage capabilities.

5. Key Mechanisms

5.1 accessModes Selection

RWO (ReadWriteOnce) : single node read/write – typical for databases such as MySQL or Kafka.

RWX (ReadWriteMany) : multiple nodes can read/write – used with NFS, CephFS, etc.

ROX (ReadOnlyMany) : multiple nodes read‑only – suitable for distributing configuration files.

Whether RWX is supported depends on the underlying storage system, not on Kubernetes itself.

5.2 ReclaimPolicy

Retain : data is kept after the PVC is deleted – recommended for critical production data.

Delete : underlying storage is automatically removed – common for cloud disks.

Recycle : deprecated and should not be used.

5.3 VolumeBindingMode (Delayed Binding)

volumeBindingMode: WaitForFirstConsumer

When set, PV creation and binding are postponed until a Pod that uses the PVC is scheduled. This ensures node‑topology alignment and is required for local storage or topology‑constrained cloud disks.

6. StatefulSet + Storage Interaction

volumeClaimTemplates:
- metadata:
    name: data
  spec:
    accessModes: ["ReadWriteOnce"]
    storageClassName: fast-ssd
    resources:
      requests:
        storage: 200Gi

Behavior:

Each Pod created by the StatefulSet receives its own PVC.

Deleting a Pod does not delete the associated PVC.

Deleting the StatefulSet also leaves the PVCs intact.

Actual data deletion follows the PV’s persistentVolumeReclaimPolicy.

7. Real‑World Incident: PVC Stuck in Pending

Problem

Using local-storage PVs.

PVC remained in Pending state.

Pod could not start.

Root Cause

The local PV was bound to a specific node.

The StorageClass did not specify volumeBindingMode: WaitForFirstConsumer.

The scheduler could not satisfy both the Pod’s node selector and the PV’s node affinity simultaneously.

Correct Configuration

provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

8. Production Storage Selection Quick Reference

MySQL / PostgreSQL : cloud disk or local SSD – RWO Kafka : local NVMe – RWO CI cache : NFS or CephFS – RWX Configuration files : ConfigMap – ROX Temporary jobs : emptyDir – no PVC needed

9. Monitoring, Expansion & Security

9.1 PVC Usage Monitoring

kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.2

Alert when less than 20 % free space remains.

9.2 Online PVC Expansion

The StorageClass must have allowVolumeExpansion: true.

The underlying filesystem must support resize (e.g., ext4, xfs).

9.3 Security Best Practices

securityContext:
  fsGroup: 1000
  runAsUser: 1000
  readOnlyRootFilesystem: true

10. Common Troubleshooting Commands

kubectl get pv,pvc,sc -A
kubectl describe pvc <em>NAME</em>
kubectl get events
kubectl exec <em>POD</em> -- df -h

Conclusion – Three Principles of Kubernetes Storage Design

1️⃣ Application‑agnostic storage implementation

2️⃣ Platform‑wide unified storage management

3️⃣ Data lifecycle independent of Pods

Kubernetes does not store data for you; it provides mechanisms to manage data safely, controllably, and recoverably.