Why Your Disk Shows Space Yet Fails: Understanding and Fixing Inode Exhaustion

1. Introduction to inodes

In Linux, file data is stored in blocks, while metadata such as owner, creation date, and size is stored in a structure called an inode (index node). During formatting the disk is divided into a data area and an inode table that holds these metadata entries.

Each inode typically occupies 128 bytes or 256 bytes, and the total number of inodes is set at format time (often one inode per 1 KB or 2 KB). For example, on a 1 GB disk with 128‑byte inodes and one inode per 1 KB, the inode table would consume about 128 MB (12.8% of the disk).

2. Inodes exhaustion

When all inodes are used, the system cannot create new files even if there is still free disk space. This usually happens when many small files fill the inode table.

Disk space may be partially used while inodes are already at 100 %.

Use df to view disk usage and df -i to view inode usage.

Both commands support the -h flag ( df -h and df -hi). In the example, disk space is at 71 % but inodes are at 100 %.

3. Solving inode exhaustion

The size of inodes is determined when the partition is formatted; larger partitions have larger inode tables.

The root partition is often small, so frequent creation of small files without timely cleanup can quickly fill inodes.

Steps to resolve inode exhaustion:

(1) Find directories with the most files for i in /*; do echo $i; find $i | wc -l; done If the range is known, replace /* with a more specific path.

The investigation revealed that /var/spool/postfix/maildrop contained many small files because cron output and warnings were being mailed, but sendmail/postfix failed, causing the files to accumulate.

A per‑minute cron job also generated a small file each minute.

(2) Delete large numbers of files

ls | xargs -n 1000 rm -rf   # use xargs to avoid deletion failures

4. Summary

(1) Add MAILTO="" as the first line in crontab -e to stop mail files from being created.

(2) Redirect output of cron jobs that do not need logs to /dev/null: */10 * * * * /tmp/test.sh >/dev/null 2>&1 (3) Periodically clean old files, e.g.:

find <directory> -type f -mtime +30 | xargs -n 1000 rm -f

(4) Monitor inode usage regularly.

Note: Pay attention to crontab syntax and ensure timely cleanup of generated files.