Do File Count and Directory Depth Really Slow Down Disk I/O on Linux?

Answer (Linux only)

File count, directory count, directory depth and their impact on disk I/O

To understand the issue you need to know how a file and a directory (collectively called a node) are represented in Linux.

File system basics

0. What is a file
1. What is a filename for
2. How to locate a file by its name
3. How directories are indexed

0. What is a file

In Linux a file is described by an inode, which is a unique identifier. The inode structure stores most of the file’s metadata.

Example of inspecting a file:

$ touch a
$ stat a
File: `a'
Size: 0          Blocks: 0   IO Block: 4096  regular empty file
Device: 801h/2049d   Inode: 786623   Links: 1
Access: (0664/-rw-rw-r--)  Uid: (1000/zb)  Gid: (1000/zb)
Access: 2014-08-21 23:07:23.311074036 -0700
Modify: 2014-08-21 23:07:23.311074036 -0700
Change: 2014-08-21 23:07:23.311074036 -0700
Birth: -

The inode number is the file’s unique key, similar to a primary key in a database. An inode typically occupies 128 KB or 256 KB, sometimes larger than the file itself. An inode stores:

1. File size
2. Physical location on disk
3. UID and GID
4. Permissions
5. Timestamps
6. Link count

Inodes can be exhausted; view inode usage with:

$ df -i
Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1      1245184 163895 1081289   14% /
udev            123823    484 123339    1% /dev
tmpfs           126215    403 125812    1% /run
none            126215      5 126210    1% /run/lock
none            126215      4 126211    1% /run/shm

If a system has many small files, many inodes are consumed, which can lead to “out of inodes” errors even when disk space remains.

1. Purpose of a filename

Filename maps to an inode

Filenames are not stored in the inode; the kernel resolves a name to an inode during operations.

2. How the kernel finds a file by name

Directories are also files

Directories have their own inode and contain dentry (directory entry) structures that point to inodes.

3. How directories are indexed

Think of an inode as a pointer

/data/shells/text.txt → / (root) dentry → inode → data dentry → inode → shells dentry → inode → text.txt inode.
Each path component is a filename stored in a dentry; each dentry points to its parent via “..”.

Impact of file and directory count

Both the number of files and directories affect the I/O required to locate a file because the directory structure itself may reside on disk. Once the file is found, additional I/O impact is minimal. Greater directory depth adds more lookup steps.

To check whether you are near a threshold, examine inode usage with df -i.