Kubernetes PV/PVC Troubleshooting: From Mount Failures to Zero Data Loss
This comprehensive guide dissects Kubernetes PV/PVC architecture conflicts, CSI mount lifecycle phases, production StorageClass baselines, and troubleshooting commands, illustrated with three real-world case studies covering Multi-Attach deadlocks, accidental PVC deletion recovery, and Terminating state deadlocks, plus emergency scripts and engineering principles for zero data loss.
1. Architecture Conflict: Stateless Orchestration vs Stateful Persistence
Kubernetes was designed around immutable infrastructure and stateless services: Pods can be killed, migrated, or recreated at any time. However, production databases (MySQL, PostgreSQL), distributed middleware (Kafka, Elasticsearch), and distributed training storage rely on underlying persistence. This creates two core conflicts:
Scheduling speed vs storage mount latency : K8s reschedules a Pod in seconds, but cloud disks/network storage (Ceph, AWS EBS, Alibaba Cloud NAS) detach and re-attach often take tens of seconds to minutes.
Multi-tenant isolation vs single-writer lock : Block storage (ReadWriteOnce) allows only single-node attachment. During network partition, K8s marks Pod Terminating and forces reschedule to a new node, but the new node hits VolumeInUse deadlock because the volume hasn't released on the old node.
The core operational task is to clarify the CSI (Container Storage Interface) mount execution chain at each stage and establish a rigorous anti-data-loss baseline.
2. CSI Mount Lifecycle Pipeline
2.1 Four Phases of PV/PVC Mount and Controller Topology
A PVC goes through coordinated processing by K8s core and CSI plugins:
Provisioning : External Provisioner watches PVC, calls cloud storage API to create real underlying volume (LUN/block device/NAS), and creates PV object.
Attaching : K8s control plane AttachDetachController watches Pod scheduled to target Node, calls CSI Controller ControllerPublishVolume to attach block device to target Node host layer (e.g., /dev/vdX).
Mounting : Target Node's kubelet (Volume Manager) calls local CSI Node plugin (DaemonSet) NodeStageVolume (format and global mount to /var/lib/kubelet/plugins/...) and NodePublishVolume (bind mount into Pod container's dedicated directory).
Running : Container starts and reads/writes data on mounted directory.
[ PVC Claim ] ---> [ PV Binding ]
|
v (K8s Control Plane)
+---------------------------+
| AttachDetachController |
+---------------------------+
|
| CSI: ControllerPublishVolume (Attach)
v
[ Node Host /dev/sdb ]
|
| CSI: NodeStageVolume (Format & Global Mount)
v
[ /var/lib/kubelet/plugins/kubernetes.io/csi/... ]
|
| CSI: NodePublishVolume (Bind Mount)
v
[ Pod Container Mount: /var/lib/mysql (VolumeMount) ]2.2 State Transition and Root Cause Matrix
Phase | Core Component | Typical Blocking Error | Common Root Cause
-----------|-------------------------|--------------------------------------------|------------------
Pending | Provisioner | WaitForFirstConsumer / FailedBinding | StorageClass error or scheduling policy delays binding
Attach | AttachDetachController | Multi-Attach error for volume | Old node offline but underlying cloud disk hasn't released Detach lock
Mount | kubelet & CSI Node | mount: /var/lib/kubelet/... is busy | Filesystem corruption, mount point leak, or stale directory
Unmount | kubelet & CSI Node | Volume is in use by Pod / device is busy | Container leftover zombie processes holding file handles3. Production Baselines and Core Configuration
3.1 Production-Grade StorageClass Baseline
Never use default StorageClass missing Retain in production. Recommended standard for CSI block storage:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-nvme-sc
provisioner: disk.csi.k8s.io
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
allowVolumeExpansion: true
parameters:
type: high-perf-ssd
fstype: ext4
encrypted: "true"3.2 Four-Dimensional Parameter Breakdown
Must-Change Items
reclaimPolicy: Retain</strong>: Production must never use <code>Delete. With Retain, even if developers accidentally delete PVC, underlying physical volume and PV object remain in Released state, preventing physical volume deletion.
volumeBindingMode: WaitForFirstConsumer</strong>: Delayed binding. Avoids PV binding to a node prematurely while Pod schedules elsewhere due to CPU/memory/taints, causing cross-node mount failure.</li></ul><p><strong>Coupled Items</strong></p><ul><li><code>allowVolumeExpansion: true</strong> with OS filesystem: Enables online expansion. Requires underlying CSI to support hot expansion; Pod restart or CSI Node auto-executes <code>resize2fs/ xfs_growfs.
fstype: ext4</strong>: Align with kernel. If using xfs, note xfs only expands, cannot shrink.</li></ul><p><strong>Performance Tuning</strong></p><p>Host mount options optimization (via <code>mountOptionsin StorageClass):
mountOptions:
- noatime
- nodiratime
- discardnoatime</strong>: Completely avoids metadata write on every file read; boosts high-concurrency small-file read/write throughput by 20%+.</li></ul><p><strong>Absolute Red Lines</strong></p><ul><li>Never modify <code>persistentVolumeReclaimPolicyof a mounted running PV without a rollback plan.
Never use non-persistent hostPath in StatefulSet as PVC substitute.
3.3 System-Level Config: Avoid Storage I/O Blocking Causing Node NotReady
cat << 'EOF' > /etc/sysctl.d/99-k8s-storage.conf
# Reduce kernel long-period D-state hangs due to slow block devices
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
# Avoid NFS/CephFS packet loss causing disconnects and long retries
net.ipv4.tcp_retries2 = 8
EOF
sysctl -p /etc/sysctl.d/99-k8s-storage.conf4. High-Frequency Commands and Troubleshooting Toolbox
4.1 Penetrate Storage Volume Attachment Chain
First step to diagnose mount stuck: locate which node still holds VolumeAttachment object:
kubectl get volumeattachment -A | grep -E "pvc-|Failed|Attached"Example output showing key insight: new node attempting attach fails because old node still holds volume.
NAME ATTACHER PV NODE ATTACHED AGE
csi-02941faeb892b10a2bb128919248 disk.csi.k8s.io pvc-6c71c429-1234-4567-89ab-cdef01234567 node-prod-02 true 42m
csi-982847281bc88192a912bb872111 disk.csi.k8s.io pvc-6c71c429-1234-4567-89ab-cdef01234567 node-prod-03 false 5m <-- [Key: attempting mount to node-03 fails, node-02 still holds volume]Extract specific Pod's underlying volume path and device mapping:
kubectl get pod mysql-0 -n storage-prod -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}'
# Output: data-mysql-04.2 Host Mount Leak Detection
On target Node, check for zombie mount points left after container kill:
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS | grep "pvc-6c71c429"Example showing Pod instance gone but mount still occupied:
/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6c71c429/globalmount /dev/vdb ext4 rw,relatime
/var/lib/kubelet/pods/e82910bb-a123/volumes/kubernetes.io~csi/pvc-6c71c429/mount /dev/vdb ext4 rw,relatime <-- [Key: Pod instance no longer exists but mount still held]Use lsof to capture orphan processes holding the directory:
lsof +D /var/lib/kubelet/pods/e82910bb-a123/volumes/kubernetes.io~csi/pvc-6c71c429/mountOutput reveals container's mysqld became orphan process on host (PPID=1), causing umount device busy.
4.3 Production Storage Monitoring PromQL
PVC Disk Usage Alert (predict full in 4 hours)
predict_linear(kubelet_volume_stats_available_bytes{namespace=~"prod-.*"}[1h], 4 * 3600) < 0Alert threshold: linear prediction < 0.
Business impact: early warning prevents MySQL/PostgreSQL entering read-only or crash.
Volume Manager Mount Failure Counter
sum by (node) (rate(storage_operation_errors_total[5m])) > 0Alert threshold: sustained > 0.
Business impact: specific node storage driver anomaly prevents new Pods from mounting volumes.
5. Deep-Water Case Studies
Case 1: Node Abnormal Crash Causes Multi-Attach Deadlock, Service Down 40 Minutes
Symptom: Production cluster node-02 suffers hardware power loss (Kernel Panic). K8s triggers node eviction after 5 minutes. Core database Pod redis-cluster-2 reschedules to node-05 but stays in ContainerCreating for long time. Investigation Commands and Evidence: Pod events show:
Warning FailedAttachVolume 3m (x12 over 15m) attachdetach-controller Multi-Attach error for volume "pvc-91823" Volume is already exclusively attached to one node and can't be attached to another
<-- [Key]: Control plane determines underlying physical disk still attached to dead node-02, refuses Attach to node-05]VolumeAttachment confirms:
csi-attach-91823 disk.csi.k8s.io pvc-91823 node-02 true 22mRoot Cause Analysis: For ReadWriteOnce (RWO) block storage, to prevent split-brain causing two nodes writing same physical disk and corrupting filesystem, K8s AttachDetachController enforces strict mutual exclusion. When node-02 loses heartbeat, K8s marks it NotReady. But control plane cannot determine if machine is fully powered off or just network partition . If it's still running and writing to disk, forcibly attaching volume to node-05 would directly cause filesystem metadata corruption. Therefore, until node-02 is confirmed detached at physical/cloud platform layer, the mutex lock never releases. Fix Procedure:
Confirm original node node-02 physically offline or IPMI powered off.
In cloud console or via cloud API, manually Force Detach the cloud disk.
Explicitly delete blocking VolumeAttachment object in K8s:
kubectl delete volumeattachment csi-attach-91823 --timeout=15sSigned-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
