Disk Full on Linux? Run These 8 Diagnostic Commands First

Problem Background

Disk‑space alerts on Linux can stem from three distinct conditions:

Space truly exhausted – df -h shows 100 % usage.

Inodes exhausted – df -h reports free space but the kernel returns No space left on device.

Deleted files still holding space – df reports full while du cannot locate large files.

Confusing these cases leads to wasted effort or accidental data loss.

Preparation Before Deletion

# Record current disk state
$ df -h > /tmp/disk_before_$(date +%s).txt
$ df -i >> /tmp/disk_before_$(date +%s).txt

# Identify the business purpose of each mount point
$ mount | grep <mount_point>
$ ls -la <mount_point> | head

If multiple data disks exist (e.g., /data, /var/log), determine which partition is full before proceeding.

Eight Commands in Detail

Command 1: df -h – View usage of each mount point

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G   38G   16M  100% /
/dev/vdb1       200G  120G   80G   60% /data

Use% = 100 % – partition is full and requires immediate action.

Use% = 90‑99 % – warning range; investigate growing directories.

Focus on the Avail column because it reflects actual free space for non‑root users; Used may include the default 5 % reserved for root. The reserve can be reduced with tune2fs -m 1 /dev/vdb1.

Command 2: df -i – Check inode consumption

$ df -i
Filesystem      Inodes IUsed IFree IUse% Mounted on
/dev/vda1      256000 255900   100  100% /

A typical trap: df -h shows free space while the system reports No space left on device. In such cases df -i often shows IUse% = 100 %, indicating inode exhaustion caused by a huge number of tiny files.

Quick ways to locate directories with many small files:

# Count files per top‑level directory (depth 1)
for dir in /*/; do
  echo -n "$dir: "
  find "$dir" -xdev -type f 2>/dev/null | wc -l
done

# More precise, recursive approach
find / -xdev -type f | awk -F/ '{print $NF=""; print $0}' | sort | uniq -c | sort -rn | head -10

Command 3: du -sh – Locate large directories layer by layer

# Show size of each first‑level directory under /
$ du -h --max-depth=1 / | sort -rh | head -10
6.2G    /usr
4.1G    /var
2.8G    /opt
1.5G    /home
1.2G    /root

# Drill into /var
$ du -h --max-depth=1 /var/ | sort -rh | head -10
3.5G    /var/log
500M    /var/lib

# Finally pinpoint a specific path
$ du -sh /var/log/nginx/
1.2G    /var/log/nginx/

du

recursively sums sub‑directories, allowing rapid narrowing down to the “culprit”.

Command 4: lsof | grep deleted – Find deleted files still held open

# List all deleted files that are still referenced
$ sudo lsof | grep deleted
COMMAND   PID USER   FD TYPE DEVICE SIZE/OFF   NODE NAME
java      3456 root   23w REG  202,1 2147483648 12345 /var/log/app/access.log (deleted)
nginx     2321 www    5w  REG  202,1 1073741824 23456 /var/log/nginx/access.log (deleted)

Key columns: SIZE/OFF – actual size of the file (e.g., 2 GB, 1 GB). NAME ending with (deleted) – file has been removed but the handle is still open. FD – file descriptor number and mode (e.g., 23w means descriptor 23 opened for writing).

Resolution steps (ordered by impact):

Notify the process to reload its file handles (most log frameworks support this):

$ kill -USR1 <PID>      # Java / log4j / syslog‑ng
$ nginx -s reopen       # Nginx
$ systemctl restart rsyslog

If the process does not respond, restart the service or force‑kill it:

# Verify the service can be restarted
$ systemctl restart <service>

# Or force kill after confirming no business impact
$ kill -9 <PID>

Temporarily free space without stopping the process by truncating the file descriptor:

$ : > /proc/<PID>/fd/<FD_NUMBER>

This clears the file content while keeping the descriptor open.

The most common cause is logrotate configurations that use create. After rotation, the old file remains open until the logging process reloads. Using copytruncate avoids this problem.

/var/log/nginx/*.log {
    daily
    rotate 30
    copytruncate   # copy then truncate, inode unchanged
    compress
    delaycompress
    missingok
    notifempty
}

Command 5: find – Search for large files

# Find files larger than 100 MB anywhere
$ find / -xdev -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | head -20

# Find files larger than 1 GB under /var
$ find /var/ -type f -size +1G -exec ls -lh {} \; 2>/dev/null

# Find files older than 7 days larger than 500 MB (good for archival cleanup)
$ find /data/archive -type f -size +500M -mtime +7 -exec ls -lh {} \; 2>/dev/null

Important options: -xdev – stay on the same filesystem. -size +100M – match files larger than the given size. -exec ls -lh {} \; – list each result with human‑readable size. 2>/dev/null – suppress permission‑denied errors.

Command 6: ncdu – Interactive disk usage explorer

# Install
$ apt install ncdu      # Debian/Ubuntu
$ yum install ncdu      # CentOS/RHEL

# Run against the root filesystem
$ ncdu /

Navigation:

Arrow keys – move up/down.

Enter – descend into a directory. d – delete the selected file or directory. q – quit. ncdu scans faster than du and updates sizes in real time.

Command 7: iostat -xz – Analyze disk I/O performance

$ iostat -xz 1 5
Linux 5.15.0-91-generic ... 08/15/2025  _x86_64_ (32 CPU)

Device   r/s   w/s   rkB/s   wkB/s  %util  r_await  w_await  aqu-sz
vda    3200   4500 128000   360000   99.8    15.20    92.30    256.0

Key metrics:

%util – disk busy percentage (warning > 80 % for HDD; less useful for SSD/NVMe).

r_await / w_await – average read/write latency (warning > 20 ms for HDD, > 2 ms for SSD).

aqu‑sz – average queue length (warning > 1 for HDD; compare with device parallelism for SSD/NVMe).

For NVMe devices, %util can be misleading; rely on aqu‑sz and latency instead.

Command 8: journalctl --disk-usage – Check systemd journal size

# Show how much space journal files occupy
$ journalctl --disk-usage
Archived and active journals use 3.8G.

# Vacuum logs older than 7 days
$ journalctl --vacuum-time=7d

# Limit total journal size to 500 M
$ journalctl --vacuum-size=500M

# Permanent limit (edit /etc/systemd/journald.conf)
[Journal]
SystemMaxUse=500M
MaxFileSec=7day

After changing the configuration, restart the service:

$ systemctl restart systemd-journald

Scenario‑Based Command Combinations

Scenario A – High space usage, du can locate the culprit

$ df -h               # Identify the full partition
$ du -h --max-depth=1 /   # Drill down level by level
$ find /path -type f -size +500M -exec ls -lh {} \;
$ ncdu /path          # Interactive confirmation

Scenario B – df shows full but du finds nothing

$ df -h
$ sudo lsof | grep deleted | head -5
$ sudo lsof | grep deleted | awk '{print $7, $NF, $2}' | sort -rn | head -10
# Send signals or restart the offending services

Scenario C – No space left on device yet df -h shows free space

$ df -i                # Confirm inode exhaustion
$ find /data -xdev -type f | wc -l   # Count files
$ find /data -xdev -type f -mtime +90 -delete   # Remove old files

Scenario D – Disk I/O poor while space is sufficient

$ iostat -xz 1 5
$ iotop -o               # Show I/O‑heavy processes
$ pidstat -d 1 5         # Per‑process I/O statistics

Production‑Ready Cleanup Workflow

Do not delete files immediately when a disk fills up. Follow this standard process:

Record the situation → Locate large directories → Confirm business impact → Backup → Clean → Verify

Step 1 – Record the situation

$ df -h > /tmp/disk_clean_$(date +%s).before
$ du -h --max-depth=1 /var/log | sort -rh > /tmp/disk_clean_$(date +%s).logdir

Step 2 – Locate and confirm

# After finding a large file, verify its business relevance
$ ls -la /var/log/nginx/access.log.20250815.gz
# Ask the owner whether it can be removed

Step 3 – Backup before cleaning

# Compress and copy to another partition
$ gzip -c /var/log/nginx/access.log.20250815.gz > /backup/nginx/access.log.20250815.gz
# Delete only after successful backup
$ rm /var/log/nginx/access.log.20250815.gz

Step 4 – Verify space release

$ df -h <mount_point>
$ df -i <mount_point>

Step 5 – Configure long‑term strategies

# logrotate example (size‑based rotation)
/var/log/nginx/*.log {
    size 500M
    rotate 14
    copytruncate
    compress
    missingok
    notifempty
}

# Crontab example for periodic cleanup
0 3 * * 0 find /data/tmp -type f -mtime +30 -delete

Preventive Measures

Configure log rotation – Ensure all log files are covered by logrotate (prefer copytruncate to avoid deleted‑file handles).

Monitoring and alerts – Tiered alerts: warning > 80 %, critical > 90 %, emergency > 95 %.

Separate partitions – Place directories like /var/log and /data on dedicated partitions to protect the root filesystem.

Regular inspections – Automate weekly scans of partition usage and growth trends of large files.

Important Notes

Do not reboot a server when the root partition is full; lack of space can prevent services from starting. %util = 100 % does not necessarily mean hardware failure; check aqu‑sz and latency first.

Significant discrepancy between df and du (normally < 5 %) warrants checking lsof | grep deleted.

Mounting over a non‑empty directory hides existing files, which still occupy space; verify with mount output.

In production, avoid blunt rm -rf; prefer truncating files with echo "" > file or truncate -s 0 file when possible.

Conclusion

The core troubleshooting path for a full disk is:

Start with df -h to identify the saturated partition.

Check df -i to rule out inode exhaustion.

If df shows full but du cannot find large files, run lsof | grep deleted.

Use layered du to locate the biggest directories.

Employ find and ncdu for precise large‑file identification.

Inspect journalctl for systemd log consumption.

Leverage iostat to exclude I/O performance bottlenecks.

Build a preventive system with proper log rotation, partition planning, and proactive monitoring to keep disks from filling up in the first place.