How to Use Kubernetes PVC for Persistent Pod Storage

Why Persistent Storage Matters

When migrating a workload to Kubernetes, recreating a Pod does not automatically restore its data. The container root filesystem, emptyDir, and image layers are unsuitable for data that must survive across Pod lifecycles. Databases, uploaded files, caches, intermediate task results, and configuration that changes over time all require a clearly defined persistence boundary and must be accessed through a PersistentVolumeClaim (PVC) .

Roles of PVC, PV, StorageClass and CSI

A PVC is merely a declaration; it does not contain a disk, perform snapshots, or handle disaster recovery. The actual data capabilities come from the backend storage system, the CSI driver, the parameters of the StorageClass, and the backup policies defined by operations. Common failure patterns such as “PVC is Bound but the application fails to start” usually stem from overlooking the node mount status, access mode, reclamation policy, or the health of the underlying volume.

Dynamic Provisioning Flow

Application creates a PVC describing size, access mode and storageClassName.

Kubernetes looks up the StorageClass to find the matching CSI provisioner.

The CSI controller creates a backend disk, filesystem, or other volume resource.

The newly created PV binds to the PVC.

When the Pod is scheduled, the kubelet uses the CSI node plugin to attach and mount the volume.

The container sees only the directory specified by volumeMounts.mountPath.

Each layer’s “success” means something different: PVC Bound only guarantees a binding relationship; PV Bound does not guarantee that the backend storage is reachable; Pod Scheduled does not guarantee a successful mount; and Container Running does not guarantee correct write permissions or data consistency.

Pre‑creation Checks

Before submitting a PVC, verify the cluster’s storage configuration:

List existing StorageClass objects and identify the default class.

Confirm that the CSI driver is installed ( kubectl get csidriver).

Check node‑level CSI plugins ( kubectl get csinode).

Ensure the target namespace has sufficient ResourceQuota for storage.

Example commands (replace <namespace> with your actual namespace):

kubectl -n <namespace> get storageclass
kubectl -n <namespace> get csidriver
kubectl -n <namespace> get csinode
kubectl -n <namespace> get resourcequota

Creating a PVC

A typical 20 GiB read‑write‑once claim looks like this (placeholders are wrapped in <...> and must be replaced with real values):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: <PVC_NAME>
  namespace: <NAMESPACE>
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: <STORAGE_CLASS_NAME>
  resources:
    requests:
      storage: 20Gi

Attach the PVC to a Deployment or StatefulSet by adding a volumeMounts entry that points to the desired mount path (e.g., /var/lib/app). Remember to set a suitable fsGroup and fsGroupChangePolicy if the CSI driver supports it.

Troubleshooting a Stuck PVC

If a PVC remains Pending, follow the ordered checklist:

Inspect the PVC object ( kubectl describe pvc <PVC_NAME>) for events such as waiting for first consumer or quota errors.

Verify the associated StorageClass parameters, especially allowVolumeExpansion and reclaimPolicy.

Check the bound PV ( kubectl get pv <PV_NAME> -o yaml) for .spec.csi.driver, .spec.csi.volumeHandle, capacity, and node affinity.

Examine Pod events for FailedScheduling, FailedAttachVolume, FailedMount, or Multi‑Attach error to pinpoint node‑topology or driver issues.

Look at VolumeAttachment objects ( kubectl get volumeattachment) and CSI controller logs for attach failures.

Do not assume that a Bound PVC means the application can read/write; always verify the mount inside the container:

kubectl -n <namespace> exec <pod_name> -- sh -c 'mount | grep /var/lib/app && df -hT /var/lib/app && stat -c "%A %U %G %u:%g %n" /var/lib/app'

Expanding a PVC

Expansion is possible only when the StorageClass has allowVolumeExpansion: true and the CSI driver supports it. The workflow is:

Confirm the current request and status: kubectl get pvc <PVC_NAME> -o yaml.

Patch the PVC with a larger size (dry‑run first):

kubectl -n <namespace> patch pvc <PVC_NAME> \
  --type=merge \
  -p '{"spec":{"resources":{"requests":{"storage":"<NEW_SIZE>"}}}}' \
  --dry-run=client -o yaml

After the dry‑run succeeds, apply the patch without --dry-run. Monitor the PVC until .status.capacity.storage reflects the new size, then verify the filesystem inside the Pod (some drivers require a Pod restart to resize the filesystem).

Static PV Binding

When an existing backend disk must be reclaimed, create a static PV that references the exact volumeHandle and set persistentVolumeReclaimPolicy: Retain. Pair it with a PVC that specifies volumeName. Example (replace placeholders):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: <PV_NAME>
spec:
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: <STORAGE_CLASS>
  claimRef:
    namespace: <NAMESPACE>
    name: <PVC_NAME>
  csi:
    driver: <CSI_DRIVER>
    volumeHandle: <BACKEND_VOLUME_ID>
    fsType: ext4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: <PVC_NAME>
  namespace: <NAMESPACE>
spec:
  volumeName: <PV_NAME>
  storageClassName: <STORAGE_CLASS>
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

Always run a server‑side dry‑run first and have the storage team verify the volumeHandle before applying.

Data Migration to a New PVC

When the storage class, zone, or access mode is wrong, create a new PVC and migrate data with a double‑mounted Pod. The migration Pod mounts both the source and target PVCs:

apiVersion: v1
kind: Pod
metadata:
  name: <migration-pod>
  namespace: <namespace>
spec:
  restartPolicy: Never
  containers:
  - name: migrate
    image: <trusted‑rsync‑image>
    command: ["sh", "-c", "sleep 86400"]
    volumeMounts:
    - name: source
      mountPath: /source
      readOnly: true
    - name: target
      mountPath: /target
  volumes:
  - name: source
    persistentVolumeClaim:
      claimName: <source-pvc>
  - name: target
    persistentVolumeClaim:
      claimName: <target-pvc>

Run a dry‑run rsync first:

kubectl -n <namespace> exec <migration-pod> -- rsync -aHAXn --numeric-ids /source/ /target/

Then perform the actual copy (without --dry-run) and, after the application is put into maintenance mode, run a second rsync with --delete to synchronize deletions if the business permits.

Deleting a PVC Safely

Because deletion can trigger irreversible data loss, follow a six‑step protection process:

Identify all Pods using the PVC and note the bound PV and its persistentVolumeReclaimPolicy.

Create a backup or CSI snapshot and verify that it is readyToUse.

Perform a dry‑run delete ( kubectl delete pvc <name> --dry-run=server) to ensure the API request is valid.

Put the workload into maintenance mode and stop all writes.

Delete the PVC and watch for Terminating status; investigate any finalizers.

If needed, recover the data by recreating a static PV (Retain) or restoring from snapshot (Delete).

Never manually remove finalizers or claimRef without confirming that the backend volume still exists.

Monitoring PVC Health

Kubelet exposes per‑PVC metrics that can be scraped by Prometheus. Example usage ratio alert:

100 * kubelet_volume_stats_used_bytes{namespace="<namespace>",persistentvolumeclaim="<PVC_NAME>"}
/ clamp_min(kubelet_volume_stats_capacity_bytes{namespace="<namespace>",persistentvolumeclaim="<PVC_NAME>"}, 1) > 80

Similarly, monitor inode usage and watch for Pending PVCs that are not in WaitForFirstConsumer mode. Combine these metrics with CSI driver health alerts for a complete view.

Common Pitfalls

Assuming a Bound PVC guarantees a functional mount.

Sharing a single RWO PVC across multiple Deployment replicas without verifying RWX support.

Changing a StorageClass to Retain and expecting existing PVs to inherit the new policy automatically.

Treating a successful snapshot as a full backup without testing restore.

Expanding a PVC without addressing underlying data growth causes.

Conclusion

Effective PVC management requires a complete evidence chain: define the persistence requirements, verify the StorageClass and CSI capabilities, create the claim, and continuously validate the binding, node attachment, mount status, and application permissions. Only then can expansion, migration, and deletion be performed safely, ensuring that PVCs remain a reliable abstraction for production data.