Disk Appears Full on CentOS Server but du Shows Much Less Usage – Root Cause and Fix

Problem: The server reports that the disk is full, but running du -sh /* shows a total far smaller than the reported usage.

System information: The machine runs CentOS 5.3. df -h shows /dev/sda3 117 GB total, 111 GB used (100%); /dev/sda1 is the boot partition.

Partition layout from fdisk -l confirms the sizes of /dev/sda1 , /dev/sda2 (swap) and /dev/sda3 .

Directory usage from du -sh /* lists the sizes of major directories (e.g., /data 2.8 GB, /home 1.6 GB, /usr 2.5 GB, /var 29 GB), whose sum is far less than the 111 GB reported by df .

Cause: While Apache and Tomcat were running, their log files were cleared. Deleting a log file that a process still has open does not free the space because the file descriptor remains allocated.

Solution: Restart Apache and Tomcat to release the held log files. After restarting Apache, df -h still shows high usage, but after restarting Tomcat the usage drops to about 38 GB (34% used).

To prevent recurrence, set up a scheduled task to delete logs older than three days, e.g.:

find /home/yang/logs/ -type f -mtime +3 | xargs rm -f

or

find /home/yang/logs/ -mtime +3 -exec rm {} \;

For the current day's log, truncate it instead of removing:

cat /dev/null > /home/yang/logs/apache_today.log

Note that simply removing access.log while Apache is writing to it only deletes the filename; the space is not reclaimed until the process is restarted.