Skip to content
Kubernetes PV/PVC Troubleshooting: From Mount Failures to Zero Data Loss
Operations 21 min read

Kubernetes PV/PVC Troubleshooting: From Mount Failures to Zero Data Loss

This comprehensive guide dissects Kubernetes PV/PVC architecture conflicts, CSI mount lifecycle phases, production StorageClass baselines, and troubleshooting commands, illustrated with three real-world case studies covering Multi-Attach deadlocks, accidental PVC deletion recovery, and Terminating state deadlocks, plus emergency scripts and engineering principles for zero data loss.

Ops Community
Ops Community
Ops Community

1. Architecture Conflict: Stateless Orchestration vs Stateful Persistence

Kubernetes was designed around immutable infrastructure and stateless services: Pods can be killed, migrated, or recreated at any time. However, production databases (MySQL, PostgreSQL), distributed middleware (Kafka, Elasticsearch), and distributed training storage rely on underlying persistence. This creates two core conflicts:

Scheduling speed vs storage mount latency : K8s reschedules a Pod in seconds, but cloud disks/network storage (Ceph, AWS EBS, Alibaba Cloud NAS) detach and re-attach often take tens of seconds to minutes.

Multi-tenant isolation vs single-writer lock : Block storage (ReadWriteOnce) allows only single-node attachment. During network partition, K8s marks Pod Terminating and forces reschedule to a new node, but the new node hits VolumeInUse deadlock because the volume hasn't released on the old node.

The core operational task is to clarify the CSI (Container Storage Interface) mount execution chain at each stage and establish a rigorous anti-data-loss baseline.

2. CSI Mount Lifecycle Pipeline

2.1 Four Phases of PV/PVC Mount and Controller Topology

A PVC goes through coordinated processing by K8s core and CSI plugins:

Provisioning : External Provisioner watches PVC, calls cloud storage API to create real underlying volume (LUN/block device/NAS), and creates PV object.

Attaching : K8s control plane AttachDetachController watches Pod scheduled to target Node, calls CSI Controller ControllerPublishVolume to attach block device to target Node host layer (e.g., /dev/vdX).

Mounting : Target Node's kubelet (Volume Manager) calls local CSI Node plugin (DaemonSet) NodeStageVolume (format and global mount to /var/lib/kubelet/plugins/...) and NodePublishVolume (bind mount into Pod container's dedicated directory).

Running : Container starts and reads/writes data on mounted directory.

[ PVC Claim ] ---> [ PV Binding ]
                 |
                 v (K8s Control Plane)
         +---------------------------+
         |  AttachDetachController   |
         +---------------------------+
                 |
                 | CSI: ControllerPublishVolume (Attach)
                 v
         [ Node Host /dev/sdb ]
                 |
                 | CSI: NodeStageVolume (Format & Global Mount)
                 v
         [ /var/lib/kubelet/plugins/kubernetes.io/csi/... ]
                 |
                 | CSI: NodePublishVolume (Bind Mount)
                 v
         [ Pod Container Mount: /var/lib/mysql (VolumeMount) ]

2.2 State Transition and Root Cause Matrix

Phase      | Core Component          | Typical Blocking Error                     | Common Root Cause
-----------|-------------------------|--------------------------------------------|------------------
Pending    | Provisioner             | WaitForFirstConsumer / FailedBinding       | StorageClass error or scheduling policy delays binding
Attach     | AttachDetachController  | Multi-Attach error for volume              | Old node offline but underlying cloud disk hasn't released Detach lock
Mount      | kubelet & CSI Node      | mount: /var/lib/kubelet/... is busy        | Filesystem corruption, mount point leak, or stale directory
Unmount    | kubelet & CSI Node      | Volume is in use by Pod / device is busy   | Container leftover zombie processes holding file handles

3. Production Baselines and Core Configuration

3.1 Production-Grade StorageClass Baseline

Never use default StorageClass missing Retain in production. Recommended standard for CSI block storage:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-nvme-sc
provisioner: disk.csi.k8s.io
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
allowVolumeExpansion: true
parameters:
  type: high-perf-ssd
  fstype: ext4
  encrypted: "true"

3.2 Four-Dimensional Parameter Breakdown

Must-Change Items

reclaimPolicy: Retain</strong>: Production must never use <code>Delete

. With Retain, even if developers accidentally delete PVC, underlying physical volume and PV object remain in Released state, preventing physical volume deletion.

volumeBindingMode: WaitForFirstConsumer</strong>: Delayed binding. Avoids PV binding to a node prematurely while Pod schedules elsewhere due to CPU/memory/taints, causing cross-node mount failure.</li></ul><p><strong>Coupled Items</strong></p><ul><li><code>allowVolumeExpansion: true</strong> with OS filesystem: Enables online expansion. Requires underlying CSI to support hot expansion; Pod restart or CSI Node auto-executes <code>resize2fs

/ xfs_growfs.

fstype: ext4</strong>: Align with kernel. If using xfs, note xfs only expands, cannot shrink.</li></ul><p><strong>Performance Tuning</strong></p><p>Host mount options optimization (via <code>mountOptions

in StorageClass):

mountOptions:
- noatime
- nodiratime
- discard
noatime</strong>: Completely avoids metadata write on every file read; boosts high-concurrency small-file read/write throughput by 20%+.</li></ul><p><strong>Absolute Red Lines</strong></p><ul><li>Never modify <code>persistentVolumeReclaimPolicy

of a mounted running PV without a rollback plan.

Never use non-persistent hostPath in StatefulSet as PVC substitute.

3.3 System-Level Config: Avoid Storage I/O Blocking Causing Node NotReady

cat << 'EOF' > /etc/sysctl.d/99-k8s-storage.conf
# Reduce kernel long-period D-state hangs due to slow block devices
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
# Avoid NFS/CephFS packet loss causing disconnects and long retries
net.ipv4.tcp_retries2 = 8
EOF
sysctl -p /etc/sysctl.d/99-k8s-storage.conf

4. High-Frequency Commands and Troubleshooting Toolbox

4.1 Penetrate Storage Volume Attachment Chain

First step to diagnose mount stuck: locate which node still holds VolumeAttachment object:

kubectl get volumeattachment -A | grep -E "pvc-|Failed|Attached"

Example output showing key insight: new node attempting attach fails because old node still holds volume.

NAME                                      ATTACHER          PV                                       NODE          ATTACHED  AGE
csi-02941faeb892b10a2bb128919248          disk.csi.k8s.io   pvc-6c71c429-1234-4567-89ab-cdef01234567   node-prod-02  true      42m
csi-982847281bc88192a912bb872111          disk.csi.k8s.io   pvc-6c71c429-1234-4567-89ab-cdef01234567   node-prod-03  false     5m <-- [Key: attempting mount to node-03 fails, node-02 still holds volume]

Extract specific Pod's underlying volume path and device mapping:

kubectl get pod mysql-0 -n storage-prod -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}'
# Output: data-mysql-0

4.2 Host Mount Leak Detection

On target Node, check for zombie mount points left after container kill:

findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS | grep "pvc-6c71c429"

Example showing Pod instance gone but mount still occupied:

/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6c71c429/globalmount /dev/vdb ext4 rw,relatime
/var/lib/kubelet/pods/e82910bb-a123/volumes/kubernetes.io~csi/pvc-6c71c429/mount /dev/vdb ext4 rw,relatime <-- [Key: Pod instance no longer exists but mount still held]

Use lsof to capture orphan processes holding the directory:

lsof +D /var/lib/kubelet/pods/e82910bb-a123/volumes/kubernetes.io~csi/pvc-6c71c429/mount

Output reveals container's mysqld became orphan process on host (PPID=1), causing umount device busy.

4.3 Production Storage Monitoring PromQL

PVC Disk Usage Alert (predict full in 4 hours)

predict_linear(kubelet_volume_stats_available_bytes{namespace=~"prod-.*"}[1h], 4 * 3600) < 0

Alert threshold: linear prediction < 0.

Business impact: early warning prevents MySQL/PostgreSQL entering read-only or crash.

Volume Manager Mount Failure Counter

sum by (node) (rate(storage_operation_errors_total[5m])) > 0

Alert threshold: sustained > 0.

Business impact: specific node storage driver anomaly prevents new Pods from mounting volumes.

5. Deep-Water Case Studies

Case 1: Node Abnormal Crash Causes Multi-Attach Deadlock, Service Down 40 Minutes

Symptom: Production cluster node-02 suffers hardware power loss (Kernel Panic). K8s triggers node eviction after 5 minutes. Core database Pod redis-cluster-2 reschedules to node-05 but stays in ContainerCreating for long time. Investigation Commands and Evidence: Pod events show:

Warning  FailedAttachVolume  3m (x12 over 15m) attachdetach-controller  Multi-Attach error for volume "pvc-91823" Volume is already exclusively attached to one node and can't be attached to another
<-- [Key]: Control plane determines underlying physical disk still attached to dead node-02, refuses Attach to node-05]

VolumeAttachment confirms:

csi-attach-91823  disk.csi.k8s.io  pvc-91823  node-02  true  22m

Root Cause Analysis: For ReadWriteOnce (RWO) block storage, to prevent split-brain causing two nodes writing same physical disk and corrupting filesystem, K8s AttachDetachController enforces strict mutual exclusion. When node-02 loses heartbeat, K8s marks it NotReady. But control plane cannot determine if machine is fully powered off or just network partition . If it's still running and writing to disk, forcibly attaching volume to node-05 would directly cause filesystem metadata corruption. Therefore, until node-02 is confirmed detached at physical/cloud platform layer, the mutex lock never releases. Fix Procedure:

Confirm original node node-02 physically offline or IPMI powered off.

In cloud console or via cloud API, manually Force Detach the cloud disk.

Explicitly delete blocking VolumeAttachment object in K8s:

kubectl delete volumeattachment csi-attach-91823 --timeout=15s
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.