Disk Full but du Shows Less? Hidden Deleted Files Held Open by Processes

Problem Scenario

At 1 AM a disk alarm triggers: df -h shows the /data partition at 98% usage. The instinctive reaction is to delete logs, but running du -sh /data/* | sort -rh | head -10 reveals that the visible directories sum to less than 50% of the partition. The remaining 40%+ is unaccounted for.

Root Cause: Deleted Files with Open File Descriptors

A process was writing to a log file. The file was removed with rm -rf, so the directory entry disappeared, but the process still holds the file descriptor. On Linux, disk space is only freed when both the directory entry and all open file handles are released. rm removes only the directory entry; the process keeps the handle, so the space remains allocated and df continues to report it as used.

Verification: Find the Ghost Files

Run the following command to list open files that have been deleted: lsof | grep deleted Typical output looks like:

java   1234 root  10w  REG  8,1  53687091200  123456  /data/logs/app.log (deleted)

The (deleted) marker confirms the file is gone from the filesystem but still held open. The size field (e.g., 53687091200 bytes ≈ 50 GB) shows exactly how much space that process is occupying.

Solutions

Option 1: Restart the Process

Restarting the Java process (or whatever holds the handle) releases the file descriptor automatically and the space is reclaimed. However, restarting a production service may not be acceptable.

Option 2: Truncate the File Descriptor In‑Place (No Restart)

Use the /proc filesystem to truncate the specific file descriptor: > /proc/1234/fd/10 Here 1234 is the PID and 10 is the file descriptor number (the number before the w in the lsof output). Executing this command instantly frees the space while the process continues running and writing to the same (now empty) file.

Prevention: Truncate Instead of Delete

To avoid this situation altogether, never rm a log file that is actively being written. Instead, truncate it with output redirection: > /data/logs/app.log The file remains in place, its content is cleared, the existing file descriptor stays valid, and the space is released immediately. This is far safer than rm.

While it is possible to clear all deleted-but-still-open files at once, the article advises against it because it can inadvertently affect unrelated processes. The recommended practice is to identify each culprit with lsof | grep deleted and handle them individually.

Summary: Three‑Step Troubleshooting Checklist

df -h

– confirm the partition is actually full. du -sh /* – verify the visible file sizes don't match the reported usage. lsof | grep deleted – catch the ghost files.

Once identified, run > /proc/pid/fd/number to reclaim the space instantly, then go back to sleep. The issue appears mysterious but is simply a matter of file‑handle management; understanding the mechanism turns a potential hours‑long outage into a three‑minute fix.