df Says Disk Full, du Says Half Empty: The Ghost File Mystery

The Midnight Disk Alert

At 1 AM a disk alarm triggers: the /data partition shows 98% used via df -h. The instinct is to delete logs, but running du -sh /data/* | sort -rh | head -10 reveals the actual files occupy less than 50%. df says full, du says half empty — where did the other 40%+ go?

Root Cause: Deleted Files Still Held Open

A process writes to a log file; you run rm -rf on that file. The directory entry disappears, but the process still holds the file descriptor. Linux only reclaims disk space when both the directory entry and all open file handles are released. rm removes only the directory entry, so the space remains allocated and df continues to report it as used.

Diagnosis: Find the Ghost Files

Run: lsof | grep deleted Typical output:

java   1234 root  10w  REG  8,1  53687091200  123456  /data/logs/app.log (deleted)

The (deleted) marker confirms the file is gone from the filesystem but still open. The size column (53687091200 bytes ≈ 50 GB) shows exactly how much space that handle occupies.

Solution 1: Restart the Process (Not Ideal for Production)

Restarting the Java process releases the handle automatically, but restarting production services is often unacceptable.

Solution 2: Truncate the File Descriptor In‑Place (No Restart)

From the lsof output, note the PID (1234) and the file descriptor number (10, from the 10w field). Then run: > /proc/1234/fd/10 This empties the file content immediately, freeing the space while the process continues writing to the same descriptor. No downtime required.

Prevention: Use Redirection Instead of rm

To avoid creating ghost files in the first place, never rm a log file that is actively being written. Instead, truncate it with: > /data/logs/app.log The file remains, its inode and open handles stay valid, and the space is released instantly.

Three‑Step Troubleshooting Checklist

df -h

— confirm the partition is actually full. du -sh /* — verify the visible file sizes. lsof | grep deleted — catch the ghost files.

Once identified, run > /proc/pid/fd/number to reclaim space in seconds. Understanding the handle mechanism turns a baffling 3 AM crisis into a three‑minute fix.