Linux Disk Space Alerts? Locate the Problem in 3 Quick Steps

Background and Purpose

Disk‑space alerts are among the most common alerts in operations. When a filesystem runs out of space, applications cannot write logs, databases cannot commit, containers cannot create new images, and even system logging may fail, leading to cascading failures. This article provides a complete workflow for locating and fixing disk‑space problems on Linux.

Prerequisites

Assumes basic familiarity with Linux commands and filesystem layout. The examples target Rocky Linux 9.4 (compatible with RHEL 9.4) using XFS. Tool versions: df (GNU coreutils 9.1), du (GNU coreutils 9.1), ncdu 1.17.

Common Causes of Disk Alerts

Log files growing rapidly

Debug mode left on (10× log volume)

Error loops that repeatedly write logs

Missing or broken log‑rotation configuration

High traffic generating massive access logs

/var/log/messages single file > 1 GB
/var/log/nginx/access.log single file > 10 GB
/var/lib/mysql/slow‑query.log continuously growing

Temporary files and caches not cleaned

/tmp accumulating files

Package manager caches (yum/dnf/apt) not cleared

Build artefacts left behind

Container images/build caches not cleaned

/var/cache occupies dozens of GB
/tmp contains files older than one month
docker system df shows unusually large usage

Data files growing naturally

MySQL InnoDB tablespace growth

PostgreSQL data directory expansion

Elasticsearch index growth

MongoDB files without automatic compression

Hidden large files

Deleted log files still held open by a process

Temporary files deleted but still written to

Interrupted image transfers leaving sparse files

df shows full, but du reports little usage
File appears deleted but space is not released
Restarting the owning process frees space

Inode exhaustion

Rare but possible; each file consumes at least one inode.

Millions of tiny cache or mail files

Filesystem created with insufficient inode count

Step 1 – Quick Identification

Basic df usage

# Show human‑readable usage
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G   45G   55G  45% /
/dev/sda2       200G  180G   20G  90% /data
/dev/sdb1       500G  450G   50G  90% /var/lib/mysql

# Show inode usage
$ df -i
Filesystem      Inode   IUsed   IFree IUse% Mounted on
/dev/sda1      524288 123456 400832   23% /
/dev/sda2     1048576 523456 525120   50% /data

# List partitions over 80% usage
$ df -h | awk 'NR>1 && $5+0 > 80 {print}'
/dev/sda2       200G   180G   20G  90% /data

Quick‑Locate Script

#!/bin/bash
# quick_disk_check.sh – locate problem within 30 seconds

echo "=== Disk Space Quick Check ==="
echo "Check time: $(date)"

echo "【1】Partitions > 80% usage:"
df -h | awk 'NR>1 && $5+0 > 80 {printf "  %s: %s/%s (Use %s)
", $1, $3, $2, $5}'

echo "【2】Overview of each mount point:"
df -h | awk 'NR>1 {printf "  %-30s %6s  Use: %s
", $6, $2, $5}'

echo "【3】Inode usage:"
df -i | awk 'NR>1 && $5+0 > 80 {printf "  Warning: %s inode usage %s
", $6, $5}'

echo "【4】Recent active directories (modified within 1 day):"
find / -maxdepth 3 -type d -mtime -1 2>/dev/null | head -20

echo "=== Quick Check Completed ==="

Step 2 – Deep Analysis

Log file analysis

#!/bin/bash
# analyze_logs.sh – inspect /var/log for large files

echo "=== Log Directory Analysis ==="

# Size of each subdirectory in /var/log
du -sh /var/log/* 2>/dev/null | sort -rh | head -15

# Files > 100 MB
find /var/log -type f -size +100M 2>/dev/null | while read f; do
  size=$(stat -c %s "$f" | numfmt --to=iec)
  mod=$(stat -c %y "$f" | cut -d' ' -f1)
  echo "  $size $mod $f"
 done

# Log rotation status
for logfile in /var/log/messages /var/log/secure /var/log/nginx/*.log; do
  [ -f "$logfile" ] && ls -lh "$logfile" | head -5
done

Database data directory analysis

#!/bin/bash
# analyze_db_space.sh – check MySQL, PostgreSQL, MongoDB sizes

MYSQL_DATA="/var/lib/mysql"
if [ -d "$MYSQL_DATA" ]; then
  echo "【1】MySQL data directory: $MYSQL_DATA"
  echo "  Total size: $(du -sh "$MYSQL_DATA" 2>/dev/null | cut -f1)"
  echo "  Top 10 tables by size:"
  find "$MYSQL_DATA" -name "*.ibd" -exec du -h {} \; 2>/dev/null | sort -rh | head -10
fi

POSTGRES_DATA="/var/lib/pgsql/data"
if [ -d "$POSTGRES_DATA" ]; then
  echo "【2】PostgreSQL data directory: $POSTGRES_DATA"
  echo "  Total size: $(du -sh "$POSTGRES_DATA" 2>/dev/null | cut -f1)"
  echo "  Base directory:"
  du -sh "$POSTGRES_DATA/base" 2>/dev/null
  echo "  WAL directory:"
  du -sh "$POSTGRES_DATA/pg_wal" 2>/dev/null
fi

MONGO_DATA="/var/lib/mongodb"
if [ -d "$MONGO_DATA" ]; then
  echo "【3】MongoDB data directory: $MONGO_DATA"
  echo "  Total size: $(du -sh "$MONGO_DATA" 2>/dev/null | cut -f1)"
  ls -lh "$MONGO_DATA"/*.wt 2>/dev/null | head -10
fi

Temporary files and cache analysis

#!/bin/bash
# analyze_temp_cache.sh – evaluate /tmp, /var/tmp and package caches

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"; }

log "【1】/tmp directory"
log "  Count: $(find /tmp -type f 2>/dev/null | wc -l)"
log "  Size: $(du -sh /tmp 2>/dev/null | cut -f1)"
find /tmp -type f -exec du -h {} \; 2>/dev/null | sort -rh | head -10

log "【2】/var/tmp directory"
log "  Count: $(find /var/tmp -type f 2>/dev/null | wc -l)"
log "  Size: $(du -sh /var/tmp 2>/dev/null | cut -f1)"
find /var/tmp -type f -exec du -h {} \; 2>/dev/null | sort -rh | head -10

log "【3】Package manager caches"
if command -v dnf >/dev/null; then
  log "DNF cache size: $(du -sh /var/cache/dnf 2>/dev/null | cut -f1)"
fi
if command -v apt-get >/dev/null; then
  log "APT cache size: $(du -sh /var/cache/apt 2>/dev/null | cut -f1)"
fi

Hidden large files

#!/bin/bash
# find_deleted_but_open.sh – locate deleted files still held open

echo "=== Deleted Files Still Holding Space ==="
for pid in $(ls /proc | grep -E '^[0-9]+$'); do
  [ -d "/proc/$pid/fd" ] || continue
  for fd in /proc/$pid/fd/*; do
    target=$(readlink "$fd" 2>/dev/null)
    [[ "$target" == *"(deleted)"* ]] && \
      size=$(ls -lh "$fd" 2>/dev/null | awk '{print $5}') && \
      cmd=$(tr '\0' ' ' < "/proc/$pid/cmdline" | cut -c1-50) && \
      echo "PID $pid | CMD $cmd | FD $fd -> $target | Size $size"
  done
done | head -50

echo "Explanation: restarting the owning process releases the space."

Docker space analysis

#!/bin/bash
# analyze_container_space.sh – Docker disk usage overview

if ! command -v docker >/dev/null; then
  echo "Docker not installed or not running"
  exit 0
fi

echo "=== Docker Space Analysis ==="

echo "【1】Overall Docker usage:"
docker system df

echo "【2】Detailed usage (containers, images, volumes, build cache):"
docker system df -v | head -50

# Dangling images
if dangling=$(docker images -f "dangling=true" -q); then
  echo "  Count: $(echo $dangling | wc -w)"
  docker images -f "dangling=true"
else
  echo "  No dangling images"
fi

# Stopped containers
if stopped=$(docker ps -a -f status=exited -q); then
  echo "  Count: $(echo $stopped | wc -w)"
  docker ps -a -f status=exited --format "{{.ID}}\t{{.Names}}\t{{.Status}}"
else
  echo "  No stopped containers"
fi

# Largest images (top 10)
docker images --format "{{.Repository}}:{{.Tag}}\t{{.Size}}" | sort -t$'\t' -k2 -rh | head -10

Step 3 – Cleanup and Preventive Alerts

Log cleanup

#!/bin/bash
# clean_old_logs.sh – delete logs older than a configurable number of days

MAX_DAYS=30
LOG_DIRS="/var/log /opt/*/logs"

echo "=== Log Cleanup ==="

deleted=0
total=0
for dir in $LOG_DIRS; do
  [ -d "$dir" ] || continue
  find "$dir" -type f -name "*.log" -mtime +$MAX_DAYS 2>/dev/null | while read file; do
    size=$(du -k "$file" | cut -f1)
    rm -f "$file"
    total=$((total+size))
    deleted=$((deleted+1))
    echo "Deleted: $file ($size KB)"
  done
  find "$dir" -type f -name "*.gz" -mtime +180 2>/dev/null | while read gz; do
    size=$(du -k "$gz" | cut -f1)
    rm -f "$gz"
    total=$((total+size))
    deleted=$((deleted+1))
    echo "Deleted: $gz ($size KB)"
  done
done

echo "Cleanup finished: $deleted files removed, approx $((total/1024)) MB freed"

Truncate current log files (keep file handles)

# Method 1: truncate
truncate -s 0 /var/log/messages

# Method 2: redirection
> /var/log/messages

# Method 3: dd (useful when the file handle must stay open)
dd if=/dev/null of=/var/log/messages

Logrotate configuration examples

# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 nginx adm
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 `cat /var/run/nginx.pid`
        fi
    endscript
}

# /etc/logrotate.d/mysql (slow query log)
/var/log/mysql/slow.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0600 mysql mysql
}

Temporary file cleanup

#!/bin/bash
# clean_temp_files.sh – purge old temporary files and caches

echo "=== Temporary File Cleanup ==="

echo "Cleaning /tmp older than 7 days"
find /tmp -type f -atime +7 -delete

echo "Cleaning /var/tmp older than 30 days"
find /var/tmp -type f -atime +30 -delete

echo "Cleaning DNF/YUM cache"
dnf clean all

echo "Cleaning old kernels (keep current and newest one)"
current=$(uname -r)
installed=$(rpm -q kernel | wc -l)
if [ $installed -gt 2 ]; then
  count=0
  for k in $(rpm -q kernel --last | sed 's/kernel-//' | cut -d' ' -f1); do
    [ $count -gt 1 ] && [ "$k" != "$current" ] && dnf remove -y "kernel-$k" && echo "Removed kernel $k"
    count=$((count+1))
  done
fi

Docker cleanup

#!/bin/bash
# clean_docker_space.sh – Docker disk usage cleanup

echo "=== Docker Space Cleanup ==="

echo "【1】Docker usage before cleanup"
 docker system df

# Remove stopped containers
stopped=$(docker ps -a -f status=exited -q | wc -l)
if [ $stopped -gt 0 ]; then
  docker container prune -f
  echo "Removed $stopped stopped containers"
else
  echo "No stopped containers"
fi

# Remove dangling images
 dangling=$(docker images -f "dangling=true" -q | wc -l)
if [ $dangling -gt 0 ]; then
  docker image prune -f
  echo "Removed $dangling dangling images"
else
  echo "No dangling images"
fi

# Remove unused volumes
unused=$(docker volume ls -f dangling=true -q | wc -l)
if [ $unused -gt 0 ]; then
  docker volume prune -f
  echo "Removed $unused unused volumes"
else
  echo "No unused volumes"
fi

echo "【2】Docker usage after cleanup"
 docker system df

Disk monitoring and alert script

#!/bin/bash
# disk_alert_monitor.sh – monitor disk usage and log alerts

WARNING_THRESHOLD=80
CRITICAL_THRESHOLD=90
LOG_FILE="/var/log/disk_alert.log"

check_disk() {
  local mp="$1"
  local usage=$(df -h "$mp" | awk 'NR==2 {gsub(/%/,"",$5); print $5}')
  local dev=$(df "$mp" | awk 'NR==2 {print $1}')
  local avail=$(df -h "$mp" | awk 'NR==2 {print $4}')
  if [ $usage -ge $CRITICAL_THRESHOLD ]; then
    echo "[CRITICAL] $mp|$dev|$usage%|$avail" >> $LOG_FILE
  elif [ $usage -ge $WARNING_THRESHOLD ]; then
    echo "[WARNING] $mp|$dev|$usage%|$avail" >> $LOG_FILE
  else
    echo "[OK] $mp|$dev|$usage%|$avail" >> $LOG_FILE
  fi
}

for mp in $(df -h | awk 'NR>1 {print $6}' | grep -vE '^(/proc|/sys|/dev/shm|/run)'); do
  check_disk "$mp"
done

Comprehensive Bash monitoring script

#!/bin/bash
# comprehensive_disk_monitor.sh – configurable monitoring with alert logging

CONFIG_FILE="/etc/sysconfig/disk_monitor.conf"
WARNING_THRESHOLD=80
CRITICAL_THRESHOLD=90
LOG_FILE="/var/log/disk_monitor.log"
EXCLUDE_MOUNTS="/proc|/sys|/dev/shm|/run"

[ -f "$CONFIG_FILE" ] && source "$CONFIG_FILE"

log(){ echo "[$1] $(date '+%Y-%m-%d %H:%M:%S') $2" >> "$LOG_FILE"; }

check_mount(){
  local mp="$1"
  local usage=$(df -h "$mp" | awk 'NR==2 {gsub(/%/,"",$5); print $5}')
  local dev=$(df "$mp" | awk 'NR==2 {print $1}')
  local avail=$(df -h "$mp" | awk 'NR==2 {print $4}')
  if [ $usage -ge $CRITICAL_THRESHOLD ]; then
    log "CRITICAL" "$mp|$dev|$usage%|$avail"
    return 2
  elif [ $usage -ge $WARNING_THRESHOLD ]; then
    log "WARNING" "$mp|$dev|$usage%|$avail"
    return 1
  else
    log "OK" "$mp|$dev|$usage%|$avail"
    return 0
  fi
}

log "========== Disk Monitoring Start =========="
mounts=$(df -h | awk 'NR>1 {print $6}' | grep -vE "^($EXCLUDE_MOUNTS)$")
crit=0; warn=0
for m in $mounts; do
  check_mount "$m"
  rc=$?
  [ $rc -eq 2 ] && crit=$((crit+1))
  [ $rc -eq 1 ] && warn=$((warn+1))
 done
log "========== Monitoring End: $crit critical, $warn warning =========="

if [ $crit -gt 0 ] || [ $warn -gt 0 ]; then
  echo "Recent alerts:"
  tail -20 "$LOG_FILE" | grep -E "CRITICAL|WARNING"
fi

LVM Logical Volume Extension

View LVM status

# Physical volumes
pvs

# Volume groups
vgs

# Logical volumes
lvs

# Detailed info
pvdisplay
vgdisplay
lvdisplay

Extend a logical volume

#!/bin/bash
# extend_lvm.sh – enlarge an LVM LV and the XFS filesystem

VG_NAME="vg00"
LV_NAME="lv_data"
EXTEND_SIZE="+50G"

echo "=== LVM Extension ==="

echo "Current LV status:"
lvs
lvdisplay /dev/${VG_NAME}/${LV_NAME}

echo "VG free space:"
vgs

echo "Extending LV by $EXTEND_SIZE"
lvextend -L $EXTEND_SIZE /dev/${VG_NAME}/${LV_NAME}

echo "Growing XFS filesystem"
xfs_growfs /dev/${VG_NAME}/${LV_NAME}

echo "Post‑extension usage:"
df -h /dev/${VG_NAME}/${LV_NAME}

Add a new disk to a volume group

#!/bin/bash
# add_disk_to_vg.sh – incorporate a new physical disk into an existing VG

NEW_PV="/dev/sdb"
VG_NAME="vg00"

echo "=== Adding New Disk to VG ==="

echo "Creating physical volume"
pvcreate "$NEW_PV"

echo "Extending volume group"
vgextend "$VG_NAME" "$NEW_PV"

echo "VG status after addition"
vgs

Disk Quota Management

Enable quota on /home (XFS)

# Edit /etc/fstab, add usrquota,grpquota to the /home line
# /dev/mapper/vg00-lv_home /home xfs defaults,usrquota,grpquota 0 0
mount -o remount /home
quotacheck -cug /home
quotaon /home

Set user quota

# edquota -u username
# Example entry (values in KB):
# Disk quotas for user username (uid 1000):
#   Filesystem   blocks   soft   hard   inodes   soft   hard
#   /dev/mapper/vg00-lv_home
#               1024    5000   6000      100      150    200

Quota reporting script

#!/bin/bash
# quota_report.sh – generate a quota usage report

echo "=== Disk Quota Report ==="
echo "Generated at: $(date)"

echo "【1】User quota summary:"
repquota -aug

echo "【2】Users exceeding quota:"
repquota -aug | grep -E '^\*' | while read line; do echo "  $line"; done

Preventive Monitoring Solutions

Zabbix agent disk items

# /etc/zabbix/zabbix_agent2.d/disk.conf
UserParameter=disk.space[*],df -h "$1" | awk 'NR>1 {gsub(/%/,"",$5); print $5}'
UserParameter=disk.inode[*],df -i "$1" | awk 'NR>1 {gsub(/%/,"",$5); print $5}'
UserParameter=disk.io[*],iostat -d "$1" | awk 'NR>4 {print $2}'
UserParameter=disk.large_files,find / -type f -size +1G 2>/dev/null | wc -l

Prometheus alert rules (disk.yml)

groups:
- name: disk
  rules:
  - alert: DiskSpaceWarning
    expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) < 0.2
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Disk space warning"
      description: "Root partition free space below 20%"
  - alert: DiskSpaceCritical
    expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) < 0.1
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Disk space critical"
      description: "Root partition free space below 10%"
  - alert: DiskInodesWarning
    expr: (node_filesystem_files_free{mountpoint="/"} / node_filesystem_files{mountpoint="/"}) < 0.1
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Inode warning"
      description: "Inode usage above 90%"

Comprehensive Bash monitoring script

#!/bin/bash
# comprehensive_disk_monitor.sh – configurable monitoring with alert logging

CONFIG_FILE="/etc/sysconfig/disk_monitor.conf"
WARNING_THRESHOLD=80
CRITICAL_THRESHOLD=90
LOG_FILE="/var/log/disk_monitor.log"
EXCLUDE_MOUNTS="/proc|/sys|/dev/shm|/run"

[ -f "$CONFIG_FILE" ] && source "$CONFIG_FILE"

log(){ echo "[$1] $(date '+%Y-%m-%d %H:%M:%S') $2" >> "$LOG_FILE"; }

check_mount(){
  local mp="$1"
  local usage=$(df -h "$mp" | awk 'NR==2 {gsub(/%/,"",$5); print $5}')
  local dev=$(df "$mp" | awk 'NR==2 {print $1}')
  local avail=$(df -h "$mp" | awk 'NR==2 {print $4}')
  if [ $usage -ge $CRITICAL_THRESHOLD ]; then
    log "CRITICAL" "$mp|$dev|$usage%|$avail"
    return 2
  elif [ $usage -ge $WARNING_THRESHOLD ]; then
    log "WARNING" "$mp|$dev|$usage%|$avail"
    return 1
  else
    log "OK" "$mp|$dev|$usage%|$avail"
    return 0
  fi
}

log "========== Disk Monitoring Start =========="
mounts=$(df -h | awk 'NR>1 {print $6}' | grep -vE "^($EXCLUDE_MOUNTS)$")
crit=0; warn=0
for m in $mounts; do
  check_mount "$m"
  rc=$?
  [ $rc -eq 2 ] && crit=$((crit+1))
  [ $rc -eq 1 ] && warn=$((warn+1))
 done
log "========== Monitoring End: $crit critical, $warn warning =========="

if [ $crit -gt 0 ] || [ $warn -gt 0 ]; then
  echo "Recent alerts:"
  tail -20 "$LOG_FILE" | grep -E "CRITICAL|WARNING"
fi

Checklist – Standard Response Procedure

【Step 1 – Within 30 seconds】
□ Run df -h, identify the problematic mount point
□ Verify that usage really exceeds the threshold

【Step 2 – Within 5 minutes】
□ Use du --max-depth=1 to find the biggest sub‑directories
□ Inspect /var/log, /tmp, /var/tmp, database data directories, Docker storage

【Step 3 – Determine cleanup method】
□ Log issue → check logrotate, truncate or delete old logs
□ Temporary files → delete files older than N days
□ Database growth → archive or purge old tables
□ Docker → docker system prune

【Step 4 – Execute cleanup】
□ Backup critical data first
□ Run the appropriate cleanup commands
□ Verify free space with df -h
□ Ensure services are still running

【Step 5 – Long‑term prevention】
□ Verify and adjust logrotate policies
□ Configure monitoring alerts (Zabbix/Prometheus)
□ Consider automated cleanup via cron
□ Document root cause for future reference

FAQ

Q: Deleted files do not free space? A: The file may still be held open by a process. Use lsof +L1 to locate the open descriptor and restart the owning process.

Q: What if inodes are full? A: Find directories with a huge number of tiny files, e.g. find / -type f -size -10k | wc -l, then clean up or consolidate those files.

Q: How to predict disk growth? A: Record daily df -h output to a time‑series database and plot trends. A simple approach is df -h >> /var/log/df_history.log and analyse the log.

Q: LVM LV is full but VG shows free space? A: Add a new physical disk, create a PV with pvcreate, extend the VG with vgextend, then enlarge the LV with lvextend and grow the filesystem.

Reference Information

Operating System: Rocky Linux 9.4

Kernel version: 6.8.5

XFS version: 1.2.1 df / du: GNU coreutils 9.1 ncdu: 1.17