Disk Full on Linux? Run These 8 Diagnostic Commands First

Problem Background

Disk‑space alerts are among the most common warnings on Linux servers. Deleting large files with rm -rf is often the first instinct, but a "disk full" condition can actually be one of three distinct situations:

Actual space exhaustion – df -h shows 100% usage.

Inode exhaustion – df -h reports free space while the kernel returns No space left on device.

Deleted files still holding space – df reports full, but du cannot locate the large files because they have been removed while a process keeps the file descriptor open.

Mixing these cases leads to wasted effort or accidental data loss.

Preparation

Before removing anything, record the current disk state and verify the business relevance of each mount point:

# Record current disk status
$ df -h > /tmp/disk_before_$(date +%s).txt
$ df -i >> /tmp/disk_before_$(date +%s).txt

# Verify mount‑point business relevance
$ mount | grep <mount_point>
$ ls -la <mount_point> | head

Eight Diagnostic Commands

Command 1: df -h – Show usage per mount point

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G   38G   16M  100% /
/dev/vdb1       200G  120G   80G   60% /data

Use% = 100% – the partition is truly full and needs immediate action.

Use% = 90‑99% – pre‑warning; investigate directories that are growing.

Check the Avail column – it reflects space available to non‑root users (root reserve is ~5% by default). The reserve can be reduced, e.g. tune2fs -m 1 /dev/vdb1.

Command 2: df -i – Check inode usage

$ df -i
Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/vda1       256000  255900      100   100% /

A typical trap: df -h shows free space but the system reports No space left on device because inodes are exhausted. Each file, regardless of size, consumes one inode; a 40 GB partition with 2.55 M tiny files can run out of inodes while using only ~10 % of the actual space.

Quickly locate directories with many files:

# Count files per top‑level directory (depth 1)
for dir in /*/; do echo -n "$dir: "; find "$dir" -xdev -type f 2>/dev/null | wc -l; done

# More precise: list directories with the most files
find / -xdev -type f | awk -F/ '{ $NF=""; print $0}' | sort | uniq -c | sort -rn | head -10

Command 3: du -sh – Locate large directories

# Top‑level sizes
$ du -h --max-depth=1 / | sort -rh | head -10
6.2G    /usr
4.1G    /var
2.8G    /opt
1.5G    /home
1.2G    /root

# Drill into /var
$ du -h --max-depth=1 /var/ | sort -rh | head -10
3.5G    /var/log
500M    /var/lib

# Final directory
$ du -sh /var/log/nginx/
1.2G    /var/log/nginx/

du

aggregates sizes recursively; iterating depth‑by‑depth quickly finds the "culprit".

Command 4: lsof | grep deleted – Find deleted but still‑open files

# List deleted files still held by processes
$ sudo lsof | grep deleted
COMMAND   PID USER FD   TYPE DEVICE SIZE/OFF   NODE NAME
java      3456 root 23w  REG  202,1 2147483648 12345 /var/log/app/access.log (deleted)
nginx     2321 www  5w  REG  202,1 1073741824 23456 /var/log/nginx/access.log (deleted)

SIZE/OFF

– actual file size in bytes. NAME ending with (deleted) – file has been removed but a descriptor remains open. FD – file‑descriptor number (e.g. 23w means write descriptor 23).

Resolution steps (recommended order):

Notify the process to reload its log file, e.g. kill -USR1 <PID> for Java/log4j, nginx -s reopen, or systemctl restart rsyslog.

If the process does not respond, restart the service or force‑kill it.

Temporarily free space without killing the process: : > /proc/<PID>/fd/<FD_NUMBER> (truncates the file).

The root cause is often logrotate using the create directive, which creates a new inode while the old file remains open. Switching to copytruncate avoids this.

Command 5: find – Search for large files

# Files >100 MB anywhere
$ find / -xdev -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | head -20

# Files >1 GB in /var
$ find /var/ -type f -size +1G -exec ls -lh {} \; 2>/dev/null

# Files >500 MB older than 7 days (archival cleanup)
$ find /data/archive -type f -size +500M -mtime +7 -exec ls -lh {} \; 2>/dev/null

-xdev

– stay on the same filesystem. -size +100M – size filter. -exec ls -lh {} – display human‑readable details. 2>/dev/null – suppress permission errors.

Command 6: ncdu – Interactive disk‑usage explorer

# Install
$ apt install ncdu   # Debian/Ubuntu
$ yum install ncdu   # CentOS/RHEL

# Run
$ ncdu /

Arrow keys to navigate.

Enter to descend. d to delete the selected entry. q to quit. ncdu scans faster than du and shows live sorted sizes.

Command 7: iostat -xz – Analyze disk I/O performance

$ iostat -xz 1 5
Linux 5.15.0-91-generic ... 08/15/2025 _x86_64_ (32 CPU)

Device   r/s   w/s   rkB/s   wkB/s  %util  r_await  w_await  aqu-sz
vda    3200  4500 128000 360000   99.8    15.20    92.30   256.0

%util

– disk busy percentage (warning >80% for HDD). r_await / w_await – average read/write latency (warning >20 ms for HDD, >2 ms for SSD). aqu-sz – average queue length (warning >1 for HDD). r/s + w/s – IOPS (≈200 for mechanical disks).

For NVMe SSDs, %util can be misleading; rely on aqu-sz and latency instead.

Command 8: journalctl --disk-usage – Check systemd journal size

# Show journal usage
$ journalctl --disk-usage
Archived and active journals use 3.8G.

# Vacuum logs older than 7 days
$ journalctl --vacuum-time=7d

# Limit journal to 500 MB
$ journalctl --vacuum-size=500M

# Permanent limit (edit /etc/systemd/journald.conf)
[Journal]
SystemMaxUse=500M
MaxFileSec=7day

After editing the config, restart the daemon: systemctl restart systemd-journald.

Scenario‑Based Command Combinations

Scenario A – High usage, du works

$ df -h
$ du -h --max-depth=1 /
$ find /path -type f -size +500M -exec ls -lh {} \;
$ ncdu /path

Scenario B – df full but du finds nothing

$ df -h
$ sudo lsof | grep deleted
# Release or restart the holding process

Scenario C – "No space left on device" yet df -h shows free space

$ df -i   # inode exhaustion confirmed
$ find /data -xdev -type f | wc -l   # count files
$ find /data -xdev -type f -mtime +90 -delete   # clean old files

Scenario D – Disk I/O bottleneck with sufficient space

$ iostat -xz
$ iotop -o   # spot I/O‑heavy processes
$ pidstat -d 1 5   # per‑process I/O stats

Standard Production Cleanup Workflow

Record state → Locate large directories → Confirm business impact → Backup → Clean → Verify

Step 1 – Record

$ df -h > /tmp/disk_clean_$(date +%s).before
$ du -h --max-depth=1 /var/ | sort -rh > /tmp/disk_clean_$(date +%s).logdir

Step 2 – Locate & Confirm

# Verify ownership of large files
$ ls -la /var/log/nginx/
# Ask the application owner before deletion

Step 3 – Backup then Delete

# Compress and move
$ gzip -c /var/log/nginx/access.log.20250815.gz > /backup/nginx/access.log.20250815.gz
# Delete after successful backup
$ rm /backup/nginx/access.log.20250815.gz

Step 4 – Verify Release

$ df -h <mount_point>
$ df -i <mount_point>

Step 5 – Long‑Term Strategy

# Logrotate example (size‑based)
/var/log/nginx/*.log {
    size 500M
    rotate 14
    copytruncate
    compress
    missingok
    notifempty
}

# Cron job for old temporary files
0 3 * * 0 find /data/tmp -type f -mtime +30 -delete

Preventive Measures

Configure log rotation; servers without logrotate will eventually fill up.

Set multi‑level alerts: Warning >80%, Critical >90%, Emergency >95% usage.

Use dedicated partitions for log and data directories to protect the root filesystem.

Run weekly automated scans of partition usage and large‑file growth.

Additional Tips

Avoid rebooting a server when the root partition is full; services may fail to start. %util = 100% does not always indicate hardware failure; check aqu-sz and latency first.

A discrepancy greater than 5% between df and du warrants checking for deleted‑but‑open files.

Mounting over a non‑empty directory hides underlying files that still consume space; verify with mount.

In production, prefer truncating a file ( truncate -s 0 file) over rm -rf to avoid losing handles.

Conclusion

The core troubleshooting path is:

Start with df -h to identify the full partition.

Check df -i to rule out inode exhaustion.

If df shows full but du cannot locate large files, run lsof | grep deleted.

Use layered du to pinpoint the biggest directories.

Employ find and ncdu for exact large files.

Inspect journalctl for systemd log consumption.

Leverage iostat to eliminate I/O performance issues.

Effective handling of a full disk is not "delete immediately" but a disciplined process of recording, confirming, backing up, cleaning, and establishing preventive monitoring and log‑rotation policies.