Kubernetes Pod Storage Panorama: Full Chain from Volume, PVC, PV to CSI

Introduction

Moving stateless services to Kubernetes is easy, but storage often blocks teams. Writing a file in Kubernetes involves questions about where the file is written, who allocates it, when it is mounted, node‑failure recovery, cross‑zone scheduling, expansion and snapshots. Treating storage as a single pipeline (Pod → Volume → PVC → PV → StorageClass → CSI) leads to incidents.

Business scenario: an e‑commerce platform

The platform handles ~18 k QPS normally and 4‑6× that during promotions on a Kubernetes 1.30 cluster. Four components are tightly coupled with storage:

order‑service (Deployment) – writes temporary files, retry queues and intermediate results.

payment‑service (Deployment) – needs shared certificates, risk‑control rules and audit archives.

MySQL master‑slave (StatefulSet) – requires low latency, scalability, backup and node‑failure recovery.

Kafka (StatefulSet) – high write throughput, order‑sensitive writes, built‑in replication.

Initial simple solution:

All temporary data on emptyDir Shared directory on NFS

MySQL using a hostPath Kafka also using a cloud disk

In production four typical problems appear:

Problem 1 – treating a temporary volume as persistent

order‑service writes image chunks to an emptyDir. When the Pod is recreated the files disappear while the DB record remains, leaving many tasks half‑failed.

Problem 2 – using a shared NFS as a universal solution

During a promotion NFS experiences short spikes; application threads block on I/O, JVM off‑heap memory grows, and many Pods are evicted or OOM‑killed.

Problem 3 – using a node‑local path as a database disk

MySQL on hostPath works while the node is healthy, but after node maintenance or crash the Pod is scheduled to a new node where the data does not exist, causing the database to fail to start.

Problem 4 – creating PVCs without understanding binding and topology

A new StatefulSet leaves its PVC Pending because the StorageClass uses immediate binding; the volume is provisioned in zone A while the Pod is scheduled to nodes in zone B, so the volume never matches the node.

Storage layers in Kubernetes

Pod declares *how* to use storage, PVC declares *how much* storage is needed, PV represents *what* storage the platform provides, StorageClass defines *how* the storage is provisioned, and CSI actually *creates and mounts* the storage.

Application layer – volumeMounts, volumes. Responsibility: application developers.

Claim layer – PersistentVolumeClaim. Responsibility: developers / middleware teams.

Supply layer – PersistentVolume. Responsibility: platform / storage teams.

Template layer – StorageClass. Responsibility: platform teams.

Plugin layer – CSI driver. Responsibility: platform / storage teams.

Node layer – kubelet + OS filesystem. Responsibility: node and container runtime.

Many people conflate Volume and PV, which is the root of misunderstanding.

Pod data lifecycle: from YAML to actual writes

storage backend → CSI Node Plugin → kubelet → CSI external‑attacher → CSI external‑provisioner → PV Controller → Scheduler → API Server → developer creates Pod + PVC → watch PVC → dynamic provisioning (CreateVolume) → volumeHandle → PV creation & binding → scheduler evaluates storage constraints → Attach (if block storage) → NodeStageVolume (device mapping, formatting, global mount) → NodePublishVolume (bind‑mount into Pod) → Pod becomes ready

Step 1 – developer submits Pod and PVC

The API Server stores the objects; no storage is attached yet.

Step 2 – PVC binding or dynamic provisioning

The persistentvolume‑controller watches PVCs. If a matching PV exists it binds immediately; otherwise, if a StorageClass is specified, the external‑provisioner calls CSI CreateVolume to create the backend volume and writes back a PV.

Step 3 – scheduler evaluates compute and storage constraints

The scheduler checks whether the volume can be attached to a node, whether the node satisfies topology constraints, whether the volume is already bound elsewhere, and whether a local disk is required. If the StorageClass uses WaitForFirstConsumer, the scheduler picks a node first, then the provisioner creates the volume in that topology.

Step 4 – Attach phase

For block storage (EBS, cloud disk, Ceph RBD) the control plane creates an VolumeAttachment via the external‑attacher. Network file systems (NFS, CephFS) are usually mounted directly without an attach step.

Step 5 – Node‑side staging and publishing

The kubelet runs the CSI node plugin: NodeStageVolume – map the device, format, perform a global mount. NodePublishVolume – bind‑mount the prepared volume into the container’s directory.

This two‑stage design lets a volume be prepared once and shared by many Pods, and makes node restarts easier to reason about.

Step 6 – Container starts and begins I/O

A Running Pod only guarantees that the mount succeeded at startup; runtime health still depends on file‑system permissions, fsGroup application, network‑FS latency and possible I/O hangs.

Common volume types and their boundaries

emptyDir – fastest to use but easy to misuse

Created when the Pod starts, deleted when the Pod is deleted.

Retains data across container restarts but not across Pod recreation.

Can be memory‑backed with medium: Memory.

Suitable for caches, transcoding intermediate files, temporary sorting and side‑car shared runtime data. Not suitable for any data that must survive Pod recreation.

hostPath – can be used but only with caution

Directly mounts a path from the host.

High performance, simple, fully bound to the node.

Significant security risk.

Good for collecting host logs, node‑level agents or quick experiments. Not suitable for production databases, portable business data or multi‑tenant workloads.

local PV – a normalized version of local disks

Managed through the PV/PVC framework, so it participates in scheduling and quota.

Supports topology constraints and clear lifecycle.

Ideal for Kafka, Elasticsearch, ClickHouse which have built‑in replication.

Drawback: if the node fails the volume does not migrate automatically; data recovery relies on the application’s own replication.

Network file system volumes (RWX) – not a universal cure

Typical implementations: NFS, CephFS, EFS, Azure Files.

Advantages: support ReadWriteMany, easy sharing, low operational overhead.

Problems: metadata operations and small‑file performance are usually worse than local or block storage; network jitter directly impacts application threads; file‑locking, directory hot‑spots and large‑file counts can become bottlenecks.

Block storage – the main battlefield for databases

Typical backends: cloud disks, Ceph RBD, SAN/LUN.

Advantages: strong random read/write performance, controllable latency, perfect for databases, KV stores and state machines.

Limitations: most expose only ReadWriteOnce; multi‑writer sharing is not native.

Production‑grade YAML examples

Order service – temporary files with emptyDir

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  labels:
    app: order-service
spec:
  replicas: 6
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
      - name: app
        image: registry.example.com/order-service:1.4.2
        ports:
        - containerPort: 8080
        env:
        - name: TMP_DIR
          value: /data/tmp
        volumeMounts:
        - name: tmp-data
          mountPath: /data/tmp
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "2"
            memory: "2Gi"
      volumes:
      - name: tmp-data
        emptyDir:
          sizeLimit: 8Gi

Two production details often missed:

Set sizeLimit on emptyDir to avoid exhausting node disk.

Separate "temporary" and "business result" directories so that transient data never overwrites persistent data.

Shared archive directory – RWX volume with a dedicated StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: shared-nfs
provisioner: nfs.csi.k8s.io
parameters:
  server: 10.10.20.15
  share: /k8s-shared
reclaimPolicy: Retain
mountOptions:
- nfsvers=4.1
- hard
- timeo=600
- retrans=2
volumeBindingMode: Immediate

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: payment-archive-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 200Gi
  storageClassName: shared-nfs

When multiple replicas write to the same directory, add two extra constraints:

Organise files per instance, e.g. /archive/${POD_NAME}/.

Make archive writes asynchronous so the main request thread does not block on the network file system.

MySQL – StatefulSet + volumeClaimTemplates + RWO block storage

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: mysql-replicapool
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  serviceName: mysql
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      securityContext:
        fsGroup: 999
      containers:
      - name: mysql
        image: mysql:8.4
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: root-password
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
        resources:
          requests:
            cpu: "1"
            memory: "2Gi"
          limits:
            cpu: "2"
            memory: "4Gi"
        readinessProbe:
          exec:
            command: ["sh", "-c", "mysqladmin ping -uroot -p$MYSQL_ROOT_PASSWORD"]
          initialDelaySeconds: 20
          periodSeconds: 10
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      storageClassName: rook-ceph-block
      resources:
        requests:
          storage: 200Gi

Engineering reasons:

StatefulSet gives a stable network identity and stable volume identity. volumeClaimTemplates guarantees a one‑to‑one mapping between Pod and PVC.

Reclaim policy Retain prevents accidental data loss when a PVC is deleted. WaitForFirstConsumer avoids topology mismatches.

Kafka – local SSD is a better fit, but you must accept node binding

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-ssd
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka-headless
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: kafka
            topologyKey: kubernetes.io/hostname
      containers:
      - name: kafka
        image: bitnami/kafka:3.8
        volumeMounts:
        - name: data
          mountPath: /bitnami/kafka
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      storageClassName: local-ssd
      resources:
        requests:
          storage: 500Gi

Key premises:

Kafka already has its own replication and ISR mechanisms.

Local disks failing are acceptable because a broker can be rebuilt after failure.

Data is not migratable between nodes; the cluster must tolerate node loss.

CSI internals: controller, node and identity services

Controller service – control‑plane actions

Common RPCs:

CreateVolume

DeleteVolume

ControllerPublishVolume

ControllerUnpublishVolume

ControllerExpandVolume

CreateSnapshot

The controller usually runs as a Deployment and works together with sidecars such as external‑provisioner, external‑attacher, etc.

Node service – actual mount on the node

Common node‑side RPCs:

NodeStageVolume

NodePublishVolume

NodeUnpublishVolume

NodeUnstageVolume

NodeExpandVolume

The node service runs as a DaemonSet because device mapping and filesystem mounting must happen locally.

Identity service – tells the world what the driver supports

GetPluginInfo

GetPluginCapabilities

Probe

Sidecars – essential parts of the CSI ecosystem

external-provisioner

– dynamic provisioning. external-attacher – creates and maintains VolumeAttachment objects. external-resizer – volume expansion. external-snapshotter – volume snapshots. node-driver-registrar – registers the node driver with kubelet. livenessprobe – health checking.

A CSI driver is therefore a set of cooperating components, not a single container.

Production‑grade CSI driver skeleton (idempotent, observable, recoverable)

package controller

import (
    "context"
    "errors"
    "fmt"
    "strings"

    "github.com/container-storage-interface/spec/lib/go/csi"
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
    "k8s.io/klog/v2"
)

type VolumeStore interface {
    FindByName(ctx context.Context, name string) (*Volume, error)
    Create(ctx context.Context, req CreateVolumeInput) (*Volume, error)
}

type Volume struct {
    ID            string
    Name          string
    CapacityBytes int64
}

type CreateVolumeInput struct {
    Name          string
    CapacityBytes int64
    Parameters    map[string]string
}

type ControllerServer struct {
    csi.UnimplementedControllerServer
    store VolumeStore
}

func (s *ControllerServer) CreateVolume(ctx context.Context, req *csi.CreateVolumeRequest) (*csi.CreateVolumeResponse, error) {
    if req.GetName() == "" {
        return nil, status.Error(codes.InvalidArgument, "volume name is required")
    }
    requiredBytes := req.GetCapacityRange().GetRequiredBytes()
    if requiredBytes <= 0 {
        return nil, status.Error(codes.InvalidArgument, "required bytes must be greater than zero")
    }
    volumeName := sanitizeVolumeName(req.GetName())
    logger := klog.FromContext(ctx)
    logger.Info("create volume requested", "name", volumeName, "requiredBytes", requiredBytes)

    existing, err := s.store.FindByName(ctx, volumeName)
    if err != nil {
        logger.Error(err, "failed to query existing volume", "name", volumeName)
        return nil, status.Error(codes.Internal, "query volume failed")
    }
    if existing != nil {
        if existing.CapacityBytes < requiredBytes {
            return nil, status.Error(codes.AlreadyExists, "existing volume capacity is smaller than requested")
        }
        logger.Info("volume already exists, return existing handle", "volumeID", existing.ID)
        return buildCreateVolumeResponse(existing), nil
    }
    volume, err := s.store.Create(ctx, CreateVolumeInput{Name: volumeName, CapacityBytes: requiredBytes, Parameters: req.GetParameters()})
    if err != nil {
        logger.Error(err, "failed to create volume", "name", volumeName)
        if errors.Is(err, context.DeadlineExceeded) {
            return nil, status.Error(codes.DeadlineExceeded, "backend create volume timeout")
        }
        return nil, status.Error(codes.Internal, "backend create volume failed")
    }
    logger.Info("volume created", "volumeID", volume.ID, "name", volume.Name)
    return buildCreateVolumeResponse(volume), nil
}

func sanitizeVolumeName(name string) string {
    name = strings.TrimSpace(name)
    name = strings.ReplaceAll(name, "/", "-")
    return fmt.Sprintf("k8s-%s", name)
}

func buildCreateVolumeResponse(v *Volume) *csi.CreateVolumeResponse {
    return &csi.CreateVolumeResponse{Volume: &csi.Volume{VolumeId: v.ID, CapacityBytes: v.CapacityBytes}}
}

Three production principles demonstrated:

Idempotent handling – repeated requests with the same name return the existing volume.

Parameter validation – never trust upstream input.

Observability – logs and error codes support troubleshooting.

In a real driver you would also implement ControllerPublishVolume, DeleteVolume, ControllerExpandVolume, node‑side NodeStageVolume and NodePublishVolume, plus metrics, tracing, timeout, retry and circuit‑breaker logic.

High concurrency and scalability: what really breaks during a promotion

Shared volume jitter amplifies into thread‑pool blockage

When an application writes directly to a shared file system in the request path, storage jitter propagates as thread‑pool blockage, connection‑pool exhaustion, upstream timeout and cascade failures.

Correct approach: decouple the main request from file writes, persist state in a database or message queue first, perform archive writes asynchronously, set timeouts and isolate the shared‑FS thread pool.

Storage expansion is not just changing PVC size

Three layers are involved: PVC spec change, CSI controller ControllerExpandVolume, and node filesystem NodeExpandVolume.

Before a production change verify that the StorageClass has allowVolumeExpansion enabled, the driver supports ControllerExpandVolume and NodeExpandVolume, and the filesystem on the node supports online expansion.

Capacity governance must be proactive

Monitor at least these signals:

PVC usage percentage, inode usage, mount errors and I/O latency.

For emptyDir also watch node ephemeral-storage pressure and per‑Pod temporary directory growth.

Data consistency and failure recovery

Persistence ≠ business consistency

Even if the volume survives, applications can see binlog written but transaction not visible, Kafka logs on disk but replicas not synced, or file written but database state not updated.

Recommended recovery workflow

Detect Pod/Volume anomaly → Is PVC still bound? → Check StorageClass & events → Does backend volume exist? → Restore from snapshot/backup → Verify node mount → Check CSI controller/node logs → Verify application data consistency → Run application‑level recovery (master‑slave switch, log replay, replica rebuild) → Restore service and monitor

Three‑layer recovery priority

Layer 1 – Volume‑level recovery (re‑attach, snapshot restore, manual PV binding).

Layer 2 – Instance‑level recovery (MySQL restore, Kafka broker restart, ES shard reallocation).

Layer 3 – Business‑level recovery (re‑run compensation jobs, replay audit archives, fix state‑file mismatches).

Observability and operational baseline

Kubernetes‑resource view

PVC Pending duration. VolumeAttachment failure events.

Pod mount‑related Warning events.

Node DiskPressure conditions.

CSI‑driver view

external-provisioner

create‑volume failure rate. external-attacher attach latency.

Node‑plugin mount failure count.

CSI sidecar resource utilisation.

Backend storage view

Volume latency, IOPS / throughput.

Remaining capacity pool.

Network‑FS connection count and metadata latency.

Application view

File‑write latency percentiles.

Archive queue backlog.

Database flush or checkpoint latency.

Task‑vs‑file state mismatch count.

Alerting only on PVC capacity provides no real storage governance.

Common pitfalls ranked by real‑world impact

Deleting a PVC with Delete reclaim policy on critical data – use reclaimPolicy: Retain, snapshots and backups.

Ignoring WaitForFirstConsumer – leads to topology mismatches (volume in zone A, pod in zone B).

Treating subPath as a universal isolation key – works differently for ConfigMap/Secret hot‑reload and can hide permission problems.

Putting network‑FS writes on the synchronous request path – storage jitter directly spikes service latency.

Forgetting fsGroup , directory permissions and container user – many "mount succeeds but app cannot start" issues are caused by wrong ownership, not a broken volume.

Reasonable evolution path – don’t chase an "all‑powerful storage platform" immediately

Small‑scale stage

Stateless apps use emptyDir.

Occasional shared data uses managed NFS/EFS.

Databases run as managed services, not in‑cluster.

Mid‑scale stage

Migrate middleware into the cluster.

Define standard StorageClasses.

Introduce CSI snapshots, expansion, monitoring.

Standardise PVC quotas and naming.

Platform‑level stage

Build a layered storage architecture.

Establish storage admission policies.

Provide per‑application storage templates.

Automate backup/restore, scaling and disaster‑recovery workflows.

The goal is to match storage solutions to workload requirements and avoid unnecessary complexity.

Conclusion: treat Pod storage as a full pipeline, not just a field

When you finally see a Pod definition like:

volumeMounts:
- name: data
  mountPath: /data

Ask yourself:

Will the data be lost?

Who creates the volume?

Will topology mismatches occur?

Can the node recover the volume after failure?

Will slow storage block application threads?

Will deleting the PVC also delete the data?

Only after answering these questions have you truly understood Kubernetes storage.