Disk Alert Panic? 10 Commands & 2 Scripts to Recover Your Server in 5 Minutes

1. Don't Start with du -sh * — It's Too Slow

When a disk alert fires at night, many admins run du -sh /var/* and wait minutes for results. On directories with many files (e.g., tens of GB of logs), du must stat every file, taking 2–3 minutes. The article recommends a tiered approach: use the right tool for the job.

Tool Comparison

ncdu — Fast, interactive, sorts by size, lets you browse like a file manager. Install via apt install ncdu (Ubuntu) or yum install ncdu (CentOS). Run ncdu /var, navigate with arrow keys, press d to delete (with confirmation). Best when you're unsure which directory is largest.

du + sort -hr — Medium speed, script-friendly, shows top N directories in one shot. Example: du -sh /var/* | sort -hr | head -10. Output lists sizes in human-readable units (K, M, G) sorted descending.

find -size — Fast for locating files exceeding a specific size. Example:

find /var -type f -size +1G -exec ls -lh {} \; | sort -k5 -hr

finds files >1GB, shows details, sorts by size (column 5). Note: many disk-full incidents are caused by "small file seas" rather than huge single files.

2. Fastest Interactive Tool: ncdu

Not installed by default but worth adding. After installation, run ncdu /var. A cursor-driven list appears, sorted by size. Enter a subdirectory, press d to delete (confirms). Ideal for exploratory analysis.

3. One-Liner for Top 10: du + sort

When ncdu isn't available (e.g., restricted internal networks), use: du -sh /var/* | sort -hr | head -10 Breakdown: du -sh /var/* — lists each item under /var with human-readable size. sort -hr — sorts descending, recognizing K/M/G suffixes. head -10 — takes the top 10 lines.

Sample output:

2.9G  /var/db
2.2G  /var/folders
2.0G  /var/vm
...

If find -size +1G returns nothing, don't panic — the culprit is often many small files. Drill down layer by layer with du or ncdu to pinpoint the cache/log directory, then clean it. Avoid blindly rm -f a single log file; it may not be the real problem.

4. Safe Deletion: Don't Just rm , Check Process Usage First

Deleting a log file with rm often fails to free space because the process still holds the file open. The directory entry is gone, but the inode and data remain allocated.

Correct Procedure

Check if a process holds the file: lsof /var/log/syslog | grep deleted If output appears, the process is still using it.

Truncate the file instead of deleting: cat /dev/null > /var/log/syslog or truncate -s 0 /var/log/syslog Space is released immediately without restarting the process.

If you must delete the file, restart the service or send kill -HUP to make the process reopen its log.

5. Scheduled Cleanup: find + Time-Based Deletion, Don't Over-Delete

Common scenario: remove logs older than 7 days.

find /var/log -name "*.log" -mtime +7 -type f -delete

-mtime +7

— modification time older than 7 days. -delete — deletes matches. Safety tip: run without -delete first to preview, or count with wc -l.

# Preview
find /var/log -name "*.log" -mtime +7 -type f | wc -l
# Confirm then execute
find /var/log -name "*.log" -mtime +7 -type f -delete

Other patterns:

By size: delete core dumps >100MB — find /core_dump -type f -size +100M -name "core.*" -delete By suffix: delete .gz archives older than 7 days —

find /var/log -name "*.gz" -mtime +7 -delete

6. One-Click Script: Disk Report + Top 10 ( disk_report.sh )

Generates a report with: overall disk usage ( df -h ), inode usage ( df -i ), top 10 directories under root (depth 2 via du -sh /* 2>/dev/null | sort -hr | head -10 ), top 10 files >500MB system-wide ( find / -type f -size +500M -exec ls -lh {} \; 2>/dev/null | awk '{print $5 "\t" $9}' | sort -hr | head -10 ), and a warning if root partition exceeds 85%. Output saved to /tmp/disk_report_YYYYMMDD.txt and printed. Optional webhook integration commented out for WeCom.

#!/bin/bash

# Disk report script
# Usage: ./disk_report.sh

REPORT_FILE="/tmp/disk_report_$(date +%Y%m%d).txt"

echo "========== Disk Usage Report ==========" > $REPORT_FILE
date >> $REPORT_FILE
echo "" >> $REPORT_FILE

# 1. Overall disk usage
echo "[Disk Partition Usage]" >> $REPORT_FILE
df -h | head -10 >> $REPORT_FILE
echo "" >> $REPORT_FILE

# 2. Inode usage (small files can exhaust inodes)
echo "[Inode Usage]" >> $REPORT_FILE
df -i | head -10 >> $REPORT_FILE
echo "" >> $REPORT_FILE

# 3. Top 10 directories under root (depth 2 to avoid full scan)
echo "[Root Top 10 Directories (Depth 2)]" >> $REPORT_FILE
du -sh /* 2>/dev/null | sort -hr | head -10 >> $REPORT_FILE
echo "" >> $REPORT_FILE

# 4. Top 10 files >500M system-wide (slow, can restrict path)
echo "[System-wide Top 10 Files (>500M)]" >> $REPORT_FILE
find / -type f -size +500M -exec ls -lh {} \; 2>/dev/null | awk '{print $5 "\t" $9}' | sort -hr | head -10 >> $REPORT_FILE
echo "" >> $REPORT_FILE

# 5. Warning items
echo "[Warnings]" >> $REPORT_FILE
DF_USAGE=$(df -h / | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $DF_USAGE -gt 85 ]; then
  echo "Root partition usage exceeds 85%: current ${DF_USAGE}%" >> $REPORT_FILE
else
  echo "Root partition usage normal: ${DF_USAGE}%" >> $REPORT_FILE
fi

echo "" >> $REPORT_FILE
echo "Report generated at: $(date)" >> $REPORT_FILE

cat $REPORT_FILE

# Optional: send to WeCom
# WEBHOOK_URL="https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxx"
# curl -s -H "Content-Type: application/json" -d "{\"msgtype\":\"text\",\"text\":{\"content\":\"$(cat $REPORT_FILE | sed ':a;N;$!ba;s/
/\
/g')\"}}" $WEBHOOK_URL

Usage: chmod +x disk_report.sh && ./disk_report.sh . Note: full scan takes time; adjust paths/permissions as needed.

7. Monitoring Script: Daily Check + Auto-Alert ( disk_alert.sh )

Lightweight version for cron. Checks root partition usage against threshold (default 85%). If exceeded, prints message and can POST to a webhook (WeCom/DingTalk/email).

#!/bin/bash

THRESHOLD=85
CURRENT=$(df -h / | tail -1 | awk '{print $5}' | sed 's/%//')

if [ $CURRENT -gt $THRESHOLD ]; then
  MESSAGE="[Disk Alert] Root partition usage reached ${CURRENT}%, exceeding threshold ${THRESHOLD}%."
  echo $MESSAGE
  # Send alert (example for WeCom)
  # curl -X POST -H "Content-Type: application/json" -d '{"msgtype":"text","text":{"content":"'$MESSAGE'"}}' YOUR_WEBHOOK_URL
fi

Add to crontab for daily 08:00 check: <code>0 8 * * * /usr/local/bin/disk_alert.sh &gt;&gt; /var/log/disk_alert.log 2&gt;&amp;1</code> 8. Case Study: /var/log Full — Root Cause Fix Production incident: /var at 98%. Culprit: /var/log/nginx/access.log at 80GB. Check process hold: lsof | grep access.log | grep deleted — no output, file not locked. Clean old logs: find /var/log/nginx -name "access.log*" -mtime +30 -delete — freed 30GB. Root cause: nginx logrotate not working. Checked /etc/logrotate.d/nginx (weekly config existed but not executing). Ran manually: logrotate -f /etc/logrotate.d/nginx . Added cron for daily logrotate. No recurrence since. Lesson: Temporary cleanup treats symptoms; configuring logrotate treats the root cause. 9. Cheat Sheet: One-Liner Reference Check disk usage: df -h Check inode usage: df -i Show largest items in current dir: du -sh * | sort -hr | head Interactive directory browser: ncdu / Find files >1GB: find / -type f -size +1G -exec ls -lh {} \; Delete logs older than 7 days: find /var/log -name "*.log" -mtime +7 -delete Truncate file without deleting: cat /dev/null &gt; file.log See which process holds a file: lsof file.log 10. Final Checklist: 5 Steps to Handle Disk Alerts df -h — confirm which partition is full. ncdu or du -sh /* | sort -hr | head — locate large directories. find with time/size filters — delete files, but preview first. If it's a log file, after cleanup configure logrotate to prevent recurrence. Deploy the monitoring script ( disk_alert.sh in cron) so you're not woken up again.