K8s Node Disk Management: Partition Planning, Mount Optimization & Monitoring Alerts

Introduction: Why Disk Management Is Critical for K8s Stability

Containerized deployments make Kubernetes node disks carry core workloads — storing container images, logs, and persistent storage. Production issues often stem from poor partition planning causing disk exhaustion, suboptimal mount parameters creating IO bottlenecks, and missing monitoring allowing failures to cascade. Modern K8s clusters (1.28+) and Containerd 1.7+ increase sensitivity to disk IO due to layered image storage and CSI dynamic provisioning, making scientific partition planning, optimized mounts, and comprehensive monitoring essential for high availability.

Core Concepts: Cross-Platform Partition Planning and LVM Expansion

Three Core Partition Principles

Isolation Principle: Separate core partitions (e.g., /var/lib/kubelet) from data and log partitions to prevent a single workload from exhausting the entire node disk.

Performance Adaptation Principle: Choose XFS for IO-intensive scenarios, ext4 for stability-focused scenarios; enable TRIM for SSDs, optimize IO schedulers for HDDs.

Extensibility Principle: Use LVM for data partitions to support online expansion; physically separate system and data disks to reduce fault propagation risk.

Production-Grade Partition Scheme (for 80GB+ Nodes)

The following table outlines recommended partitions, sizes, filesystems, purposes, and differentiated alert thresholds:

/ (root): 50GB, ext4, system core files and basic tools — 80% warning / 90% critical.

/var/lib/kubelet: 100GB+, XFS, kubelet working directory and pod ephemeral storage — 85% warning / 95% critical.

/var/lib/containerd: 200GB+, XFS, container image storage and layered data — 85% warning / 95% critical.

/var/log: 80GB, ext4, system and container logs — 70% warning / 85% critical.

/data: remaining capacity, XFS, persistent storage mount points and application data — 85% warning / 95% critical.

/tmp: 20GB, tmpfs (memory-backed), temporary files to reduce disk IO — 90% warning / 95% critical.

LVM Online Expansion Procedure (Cross-Platform)

Example: adding a new disk /dev/sdc to expand the /data partition:

# 1. Create physical volume
pvcreate /dev/sdc

# 2. Find volume group name (assume vg_data for /data)
vgdisplay | grep "VG Name"

# 3. Add physical volume to volume group
vgextend vg_data /dev/sdc

# 4. Find logical volume name (assume lv_data)
lvdisplay | grep "LV Name"

# 5. Extend logical volume to use all free space
lvextend -l +100%FREE /dev/vg_data/lv_data

# 6. Grow filesystem online (XFS uses xfs_growfs, ext4 uses resize2fs)
xfs_growfs /data  # XFS
# resize2fs /data  # ext4 (requires unmount; migrate workloads first in production)

Practice: Cross-Platform Mount Configuration and CSI Adaptation

Basic Mount Configuration Across Distributions

CentOS 9 / RHEL 9 (mount /dev/sdb to /var/lib/containerd)

# Install XFS tools if missing
yum install -y xfsprogs

# Partition with GPT
parted /dev/sdb mklabel gpt mkpart primary 0% 100%

# Format with reflink=1 for image storage optimization
mkfs.xfs -m reflink=1 /dev/sdb1

# Persistent mount in /etc/fstab
echo "/dev/sdb1 /var/lib/containerd xfs defaults,noatime,nodiratime,logbsize=256k 0 2" >> /etc/fstab

# Apply configuration
mount -a && systemctl daemon-reload

Debian 12 / Ubuntu 22.04 (Key Differences)

# Install XFS tools (package name differs)
apt install -y xfsprogs

# Partition using fdisk (beginner-friendly)
fdisk /dev/sdb << EOF
g
n
1


w
EOF

# Format and mount (same as CentOS after this)
mkfs.xfs -m reflink=1 /dev/sdb1
mkdir -p /var/lib/containerd
echo "/dev/sdb1 /var/lib/containerd xfs defaults,noatime,nodiratime,logbsize=256k 0 2" >> /etc/fstab
mount -a

Mount Parameter Optimization Details

noatime/nodiratime: Disable access time updates, reducing IO write pressure (universal optimization).

logbsize=256k: Increase XFS log buffer size, boosting metadata operation efficiency (significant for image pulls).

discard: Enable TRIM on SSDs (add to mount options), disable on HDDs to avoid performance penalty.

Local PV and Ceph CSI StorageClass Configuration

Local PV StorageClass (for local disk partitions)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-pv-highperf
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer  # delayed binding to match node disks
reclaimPolicy: Retain
mountOptions:
  - noatime
  - nodiratime
allowedTopologies:
  - matchLabelExpressions:
      - key: node-role.kubernetes.io/worker
        values:
          - storage-node

Ceph CSI StorageClass (optimized mount performance)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-csi-rbd
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: ceph-cluster-id
  pool: k8s-pool
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: ceph-admin-secret
  csi.storage.k8s.io/node-stage-secret-name: ceph-node-secret
  mounter: kernel  # kernel mount mode outperforms fuse
mountOptions:
  - noatime
  - nodiratime
reclaimPolicy: Delete
allowVolumeExpansion: true

Monitoring and Alerting: Full-Stack Visibility with Hardware Monitoring

Core Metrics Collection and Grafana Setup

Use node-exporter (1.6+) with smartmontools to collect disk metrics. Key monitoring items:

Basic metrics: node_filesystem_usage_percent (partition usage), node_disk_io_time_seconds_total (IO time).

Hardware metrics: smartmon_device_health_status (health status), smartmon_device_temperature_celsius (temperature).

Key PromQL Queries

# 1. Partition usage (filter by mountpoint)
100 * (1 - (node_filesystem_avail_bytes{mountpoint=~"/var/lib/containerd|/data"} / node_filesystem_size_bytes{mountpoint=~"/var/lib/containerd|/data"}))

# 2. Disk health status (0=normal, 1=abnormal)
smartmon_device_health_status{device=~"/dev/sd.*"}

# 3. Disk IO utilization
sum(rate(node_disk_io_time_seconds_total[5m])) by (device) / sum(node_disk_io_time_seconds_total{mode="idle"}[5m]) by (device) * 100

Grafana Dashboard Core Panels

Panel 1: Partition usage gauge (grouped by mountpoint, differentiated threshold colors).

Panel 2: Disk IO latency trend (grouped by device, overlay IO wait time).

Panel 3: Hardware health list (red highlight for abnormal disks, show temperature).

Differentiated Alerting Rules

groups:
- name: disk_alerts
  rules:
  # Log partition alert (lower threshold)
  - alert: LogDiskUsageHigh
    expr: 100 * (1 - (node_filesystem_avail_bytes{mountpoint="/var/log"} / node_filesystem_size_bytes{mountpoint="/var/log"})) > 70
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Log partition usage high"
      description: "Node {{ $labels.instance }} {{ $labels.mountpoint }} partition usage reached {{ $value | humanizePercentage }}"

  # Core partition critical alert
  - alert: CoreDiskUsageCritical
    expr: 100 * (1 - (node_filesystem_avail_bytes{mountpoint=~"/var/lib/containerd|/var/lib/kubelet"} / node_filesystem_size_bytes{mountpoint=~"/var/lib/containerd|/var/lib/kubelet"})) > 95
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Core partition usage critical"
      description: "Node {{ $labels.instance }} {{ $labels.mountpoint }} partition usage reached {{ $value | humanizePercentage }}, immediate action required"

  # Disk hardware failure alert
  - alert: DiskHardwareFailure
    expr: smartmon_device_health_status{device=~"/dev/sd.*"} == 1
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Disk hardware failure"
      description: "Node {{ $labels.instance }} {{ $labels.device }} disk health abnormal, possible bad sectors or hardware damage"

  # IO performance alert
  - alert: DiskIoHigh
    expr: sum(rate(node_disk_io_time_seconds_total[5m])) by (instance, device) / sum(node_disk_io_time_seconds_total{mode="idle"}[5m]) by (instance, device) * 100 > 90
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Disk IO utilization high"
      description: "Node {{ $labels.instance }} {{ $labels.device }} disk IO utilization reached {{ $value | humanizePercentage }}"

Best Practices and Emergency Troubleshooting

Automated Operations and Risk Control

Containerd Image Automated Cleanup (CronJob)

apiVersion: batch/v1
kind: CronJob
metadata:
  name: containerd-image-prune
  namespace: kube-system
spec:
  schedule: "0 3 * * *"  # 3 AM (off-peak)
  jobTemplate:
    spec:
      template:
        spec:
          hostPID: true  # requires host PID namespace
          containers:
          - name: prune
            image: containerd/containerd:v1.7.11
            command: ["/bin/sh", "-c"]
            args:
            - |
              # Retain images used by running pods
              running_images=$(crictl ps -q | xargs crictl inspect --output go-template --template '{{range .status.containerStatuses}}{{.imageRef}}{{\"
\"}}{{end}}' | sort -u)
              # Prune images unused for 30 days, keep latest 10, exclude running images
              crictl rmi --prune --keep-latest 10 --older-than 720h $(crictl images -q | grep -v -F "$running_images")
            volumeMounts:
            - name: containerd-sock
              mountPath: /run/containerd/containerd.sock
          volumes:
          - name: containerd-sock
            hostPath:
              path: /run/containerd/containerd.sock
          restartPolicy: OnFailure

Emergency Troubleshooting Procedures

Sudden Disk Usage Spike Investigation

# 1. Find large files (exclude virtual filesystems like /proc)
du -sh / --exclude=/proc 2>/dev/null | sort -rh | head -20

# 2. Locate deleted but still open files (hidden large files)
lsof | grep deleted | grep -E "(log|tar|tmp)"

# 3. Identify owning process and release (cautiously, verify process safety)
kill -15 <PID>  # graceful termination

# 4. Emergency cleanup of dangling Containerd images
crictl rmi $(crictl images -q --filter "dangling=true")

Filesystem Failure Emergency Repair

For core partition (e.g., /var/lib/containerd) failure, avoid direct unmount repair; follow this flow:

Mark node unschedulable: kubectl cordon <node-name> Evict pods from node:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Unmount and repair: umount /var/lib/containerd && xfs_repair /dev/sdb1 Remount and restore node:

mount /var/lib/containerd && kubectl uncordon <node-name>

Disk IO Hang Investigation

# 1. Check kernel IO error messages
dmesg | grep -i "io error\|sd[a-z]"

# 2. Trace disk IO requests (requires blktrace)
blktrace -d /dev/sdb -o - | blkparse -i -

# 3. Identify high-IO processes
pidstat -d 1  # real-time process IO
iotop  # interactive IO top (install if needed)

# 4. Check Pod resource limits (missing IO limits may cause abuse)
kubectl describe pod <pod-name> | grep -A 10 "Resources"

Conclusion

K8s node disk management centers on "full-scenario adaptation, advanced capability implementation, monitoring safety net, and controllable emergency response." Cross-platform partition planning addresses heterogeneous environment pain points; LVM and CSI adaptation meet dynamic scaling needs; automated cleanup and differentiated monitoring reduce operational overhead; robust emergency procedures ensure rapid fault recovery.

In practice, strategies must be dynamically tuned to business characteristics: AI training workloads with large files may increase /data capacity and enable LVM thin provisioning; high-frequency image update scenarios can optimize Containerd cleanup cycles; financial core workloads require enhanced hardware monitoring and multi-node redundancy. Adhering to these best practices effectively lowers disk-related failure rates, ensuring long-term stable operation of Kubernetes clusters.