Practical Guide to Diagnosing and Resolving Linux Disk Space Exhaustion

Overview

Linux file systems store data in blocks (the smallest unit of on‑disk data) and metadata in inodes . ext4 uses 4 KB blocks by default; xfs behaves similarly. Each file system allocates a fixed number of inodes at format time (ext4 ~1 inode per 16 KB, xfs allocates dynamically). Disk‑full situations can be classified into five categories:

Block exhaustion – large files consume all data blocks (e.g., an unchecked log growing to dozens of gigabytes).

Inode exhaustion – a huge number of small files fill the inode table (e.g., mail queues, session caches).

Deleted files still open – a process holds a file descriptor to a file that has been removed; df and du diverge.

Reserved space – ext4 reserves 5 % of space for the root user, which can waste tens or hundreds of gigabytes on large data volumes.

Filesystem corruption or abnormal mounts – damaged superblocks or a mount that hides existing data.

Technical Stack

Filesystem layer: ext4 extent tree, xfs allocation groups, btrfs copy‑on‑write.

VFS layer: inode cache, dentry cache, file‑descriptor management.

Process layer: /proc/<pid>/fd and lsof.

Block‑device layer: LVM, RAID, partition tables.

Applicable Scenarios

Production servers where disk usage exceeds an 80 % alert threshold.

Deployment failures reporting “No space left on device”.

Periodic inspections that reveal abnormal growth.

Container hosts whose root filesystem is filled by containers.

Database services that cannot write because the disk is full.

CI/CD pipelines that accumulate large build artefacts.

Environment Requirements

OS: Ubuntu 24.04 LTS or Rocky Linux 9.5 (kernel 6.12+).

Core utilities: util‑linux ≥ 2.40, coreutils ≥ 9.5, lsof ≥ 4.99, ncdu ≥ 2.4, logrotate ≥ 3.22, prometheus‑node_exporter ≥ 1.9.

Step‑by‑Step Procedure

1. Preparation

Gather a global overview of the host:

# Show OS name and version
cat /etc/os-release | grep -E "^(NAME|VERSION)="

# Kernel version
uname -r

# All mount points with filesystem type
df -Th

# Inode usage per filesystem
df -i

# Block‑device topology
lsblk -f

Typical df output shows partitions approaching 100 % usage. Note that the Avail column is smaller than Size - Used because ext4 reserves space for root.

2. Install Diagnostic Tools

# Ubuntu 24.04
sudo apt update && sudo apt install -y lsof ncdu sysstat iotop-c tree

# Rocky Linux 9.5
sudo dnf install -y lsof ncdu sysstat iotop tree

ncdu

2.x is a Rust rewrite and is 3‑5 × faster on directories containing millions of files.

3. Quick Type Detection Script

#!/bin/bash
DEVICE=${1:-"/dev/sda3"}
MOUNT=$(findmnt -n -o TARGET "$DEVICE" 2>/dev/null)
if [ -z "$MOUNT" ]; then echo "Device $DEVICE not mounted"; exit 1; fi

BLOCK_USE=$(df --output=pcent "$MOUNT" | tail -1 | tr -d ' %')
INODE_USE=$(df --output=ipcent "$MOUNT" | tail -1 | tr -d ' %')
DF_USED=$(df -B1 --output=used "$MOUNT" | tail -1 | tr -d ' ')
DU_USED=$(sudo du -sb "$MOUNT" 2>/dev/null | awk '{print $1}')
DIFF=$((DF_USED - DU_USED))
DIFF_MB=$((DIFF / 1024 / 1024))

if [ $BLOCK_USE -ge 95 ] && [ $INODE_USE -lt 80 ]; then
  echo ">>> Diagnosis: Block space exhausted – clean large files"
elif [ $INODE_USE -ge 95 ] && [ $BLOCK_USE -lt 80 ]; then
  echo ">>> Diagnosis: Inode exhaustion – clean many small files"
elif [ $DIFF_MB -gt 1024 ]; then
  echo ">>> Diagnosis: Deleted but unreleased files (~${DIFF_MB} MB)"
elif [ $BLOCK_USE -ge 95 ] && [ $INODE_USE -ge 95 ]; then
  echo ">>> Diagnosis: Both block and inode exhausted"
else
  echo ">>> Diagnosis: Disk status normal"
fi

4. Core Investigation

4.1 df/du/lsof Three‑Pronged Approach

df – locate partitions with >80 % usage.

du – drill down the directory tree, e.g.

sudo du -sh /* --exclude={/proc,/sys,/dev,/run} | sort -rh | head -15

lsof +L1 – list deleted files that are still held open; sort by size to find the biggest offenders.

sudo lsof +L1 2>/dev/null | grep -i deleted | awk '$7 ~ /^[0-9]+$/ {print $7,$1,$2,$9}' | sort -rn | head -20

Example output shows a 32 GB Java process holding a deleted log file.

4.2 Handling Deleted Files

Restart the process – the most reliable way to release the space.

Truncate the file descriptor – atomic and safe: sudo truncate -s 0 /proc/12345/fd/15 Signal the process to reopen its logs – e.g. kill -USR1 $(cat /run/nginx.pid) for Nginx.

4.3 Large‑File Discovery

# Find files larger than 1 GB
sudo find / -xdev -type f -size +1G -exec ls -lh {} \; | sort -k5 -rh | head -10

# Find files larger than 500 MB modified in the last 24 h
sudo find / -xdev -type f -size +500M -mtime -1 -printf '%s %T+ %p
' | sort -rn | head -10

The -xdev flag restricts the search to the current filesystem and avoids crossing into /proc or /sys.

4.4 Log‑Growth Mitigation

Immediate actions for an exploding log file:

Truncate the active log file without losing the file descriptor: sudo truncate -s 0 /var/log/app/application.log Keep only the last 1 000 lines:

sudo tail -n 1000 /var/log/app/application.log > /tmp/tail.log && sudo cp /tmp/tail.log /var/log/app/application.log && rm /tmp/tail.log

Compress rotated logs: sudo gzip /var/log/app/application.log.1 Long‑term solution – a robust logrotate configuration (see Section 4.5). Example rotates daily, keeps 14 files, compresses with gzip, and sends SIGHUP to the service after rotation.

4.5 Inode Exhaustion

When df -h shows free space but file creation fails, check inode usage with df -i. To locate the inode‑heavy directory:

# Count files per top‑level directory
for dir in /*; do
  [ -d "$dir" ] || continue
  count=$(sudo find "$dir" -xdev -type f 2>/dev/null | wc -l)
  echo "$count $dir"
done | sort -rn | head -10

Typical output points to a directory such as /var/spool. Clean up with batch find … -mtime +30 -print0 | xargs -0 -r rm -f or, for millions of files, delete in chunks ( -n 10000) to avoid I/O storms.

4.6 Reserved Space Adjustment

ext4 reserves 5 % of space for root. On pure data partitions this can be reduced:

# Show current reservation
sudo tune2fs -l /dev/sdb1 | grep -i reserved

# Reduce to 1 %
sudo tune2fs -m 1 /dev/sdb1

# Or set to 0 % for non‑system data
sudo tune2fs -m 0 /dev/sdb1

On a 2 TB ext4 volume, changing from 5 % to 1 % frees roughly 80 GB instantly.

Best Practices & Cautions

Performance Optimisation

I/O scheduler selection – kernel 6.12 defaults to mq-deadline or bfq. Use none (direct‑dispatch) for NVMe SSDs; use mq-deadline for SATA SSDs.

# View current scheduler
cat /sys/block/sda/queue/scheduler

# Temporary change (lost after reboot)
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

# Persistent change via udev rule
cat > /etc/udev/rules.d/60-io-scheduler.rules <<'EOF'
# NVMe SSD – none
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"
# SATA SSD – mq-deadline
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"
# HDD – bfq
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"
EOF
sudo udevadm control --reload-rules

Mount options – disable atime updates and tune commit intervals:

# /etc/fstab example for ext4
/dev/sdb1  /data  ext4  defaults,noatime,commit=60  0 2

# /etc/fstab example for xfs
/dev/sdc1  /storage  xfs  defaults,noatime,logbufs=8,logbsize=256k  0 2

Periodic TRIM for SSDs – enable the systemd timer:

sudo systemctl enable --now fstrim.timer
sudo fstrim -av

Security Hardening

Enforce user quotas (e.g. setquota -u appuser 50G 60G 0 0 /data).

Configure systemd‑tmpfiles to clean /tmp and other temporary directories.

Limit Docker overlay2 growth with docker system df -v and set log‑opts max-size=100m in /etc/docker/daemon.json.

High Availability

LVM thin provisioning – create a thin pool and thin LV, enable auto‑extend thresholds (default 80 % usage triggers a 20 % growth).

# Thin pool (500 GB)
sudo lvcreate -L 500G --thinpool thinpool vg_data
# Thin LV (virtual size 1 TB)
sudo lvcreate -V 1T --thin -n app_data vg_data/thinpool
# Verify usage
sudo lvs -o+lv_layout,pool_lv,data_percent,metadata_percent vg_data

Automation script that expands a logical volume when usage exceeds a threshold (example in Section 5.3).

Monitoring & Alerting

Prometheus Node Exporter

Deploy node_exporter v1.9+ to collect node_filesystem_* and node_filesystem_files_* metrics. Exclude virtual filesystems in the service definition.

Alert Rules (Prometheus)

DiskSpaceWarning – trigger when free space falls below 20 % (used > 80 %).

DiskSpaceCritical – trigger when free space falls below 10 % (used > 90 %).

InodeUsageHigh – trigger when inode usage exceeds 80 %.

DiskWillFillIn7Days – use predict_linear on the node_filesystem_avail_bytes series to warn if the trend predicts exhaustion within a week.

DiskIOHigh – trigger when rate(node_disk_io_time_seconds_total[5m]) * 100 exceeds 80 % for 15 minutes.

Grafana Dashboard

A sample dashboard visualises per‑mount usage gauges (green < 75 %, yellow 75‑85 %, orange 85‑95 %, red > 95 %), space‑trend time series, inode usage, and I/O utilisation.

Backup & Recovery

Configuration Backup Script

#!/bin/bash
set -euo pipefail

BACKUP_DIR="/var/backup/disk_config"
DATE=$(date '+%Y%m%d')
mkdir -p "${BACKUP_DIR}/${DATE}"

# Partition tables (MBR & GPT)
for disk in $(lsblk -dno NAME | grep -v loop); do
  sudo sfdisk -d "/dev/${disk}" > "${BACKUP_DIR}/${DATE}/${disk}_partition.dump" 2>/dev/null
  sudo sgdisk --backup="${BACKUP_DIR}/${DATE}/${disk}_gpt.bin" "/dev/${disk}" 2>/dev/null
done

# LVM metadata
sudo vgcfgbackup -f "${BACKUP_DIR}/${DATE}/lvm_vg_%s.conf" 2>/dev/null
sudo pvs > "${BACKUP_DIR}/${DATE}/pvs.txt"
sudo vgs > "${BACKUP_DIR}/${DATE}/vgs.txt"
sudo lvs > "${BACKUP_DIR}/${DATE}/lvs.txt"

# fstab
cp /etc/fstab "${BACKUP_DIR}/${DATE}/fstab.bak"

# Filesystem‑specific parameters
for dev in $(df --output=source | tail -n +2 | grep '^/dev'); do
  fstype=$(df -T "$dev" | tail -1 | awk '{print $2}')
  devname=$(basename "$dev")
  if [ "$fstype" = "ext4" ]; then
    sudo tune2fs -l "$dev" > "${BACKUP_DIR}/${DATE}/${devname}_tune2fs.txt" 2>/dev/null
  elif [ "$fstype" = "xfs" ]; then
    sudo xfs_info "$(df --output=target "$dev" | tail -1)" > "${BACKUP_DIR}/${DATE}/${devname}_xfs_info.txt" 2>/dev/null
  fi
done

# logrotate configuration
cp -r /etc/logrotate.d/ "${BACKUP_DIR}/${DATE}/logrotate.d/"
cp /etc/logrotate.conf "${BACKUP_DIR}/${DATE}/logrotate.conf"

# Retain 90 days of backups
find "${BACKUP_DIR}" -maxdepth 1 -type d -mtime +90 -exec rm -rf {} \;

echo "Backup completed: ${BACKUP_DIR}/${DATE}/"
ls -la "${BACKUP_DIR}/${DATE}/"

Recovery Steps

Restore partition tables:

# MBR
sudo sfdisk /dev/sda < /var/backup/disk_config/20260313/sda_partition.dump
# GPT
sudo sgdisk --load-backup=/var/backup/disk_config/20260313/sda_gpt.bin /dev/sda
# Notify kernel
sudo partprobe /dev/sda

Restore LVM configuration:

sudo vgcfgrestore -f /var/backup/disk_config/20260313/lvm_vg_vg_data.conf vg_data
sudo vgchange -ay vg_data

Restore /etc/fstab and remount:

sudo cp /var/backup/disk_config/20260313/fstab.bak /etc/fstab
sudo mount -a

Verify the restored state:

lsblk -f
df -Th
mount | grep -v cgroup

Case Studies

Case 1 – Java Application Log Exhaustion

Scenario: A Java microservice generated a 213 GB stdout.log because log rotation was not configured.

Investigation:

# Verify disk pressure
df -Th /var

# Locate the huge file
sudo find /var -xdev -type f -size +10G -exec ls -lh {} \;

# Estimate daily growth (213 GB / 88 days ≈ 2.4 GB/day)

Remediation: Keep the last 10 000 lines, truncate the file, and deploy a logrotate configuration (see Section 4.4). After truncation, disk usage dropped from 97 % to 12 %.

Case 2 – Deleted Nginx Log Still Consuming Space

Scenario: An operator removed /var/log/nginx/access.log with rm. df still reported 97 % usage.

Investigation:

# Compare du and df
sudo du -sh /var

df -Th /var

# Find deleted open files
sudo lsof +L1 | grep nginx

Output showed a 32 GB deleted file held by PID 1234.

Remediation: Either restart Nginx or truncate the descriptor:

sudo truncate -s 0 /proc/1234/fd/5
sudo truncate -s 0 /proc/1234/fd/6
# Or signal Nginx to reopen logs
sudo kill -USR1 $(cat /run/nginx.pid)

After the fix, df reported 13 % usage and lsof +L1 returned no results.

Case 3 – Inode Exhaustion by PHP Session Files

Scenario: A PHP application accumulated 6.5 million session files under /var/lib/php/sessions/, causing inode usage to reach 100 %.

Investigation:

# Confirm inode exhaustion
df -i /

# Identify the inode‑heavy directory
for dir in /var/lib/*; do
  [ -d "$dir" ] || continue
  cnt=$(sudo find "$dir" -xdev -type f 2>/dev/null | wc -l)
  echo "$cnt $dir"
done | sort -rn | head -5

Result: /var/lib/php held >5 million files.

Remediation: Batch delete old session files and adjust PHP’s garbage‑collection settings:

# Delete files older than 30 days in batches of 5 000
sudo find /var/lib/php/sessions/ -type f -mtime +30 -print0 | xargs -0 -r -n 5000 rm -f

# Adjust php.ini
session.gc_probability = 1
session.gc_divisor = 100
session.gc_maxlifetime = 1440

After cleanup, inode usage fell to 5 % and the application resumed normal operation.

Conclusion

The workflow presented consolidates the essential steps for diagnosing and remediating Linux disk‑space emergencies:

Classify the problem with a quick script.

Use df, du and lsof +L1 to pinpoint block, inode or deleted‑file issues.

Apply targeted remediation – truncate, restart, logrotate, inode cleanup, or reserved‑space adjustment.

Automate monitoring with node_exporter, Prometheus alerts and Grafana dashboards.

Maintain backup of partition tables, LVM metadata and logrotate configuration for rapid recovery.

Mastering these commands, scripts and best‑practice configurations enables operators to prevent outages, optimise storage utilisation, and keep Linux systems stable under heavy load.