Why lsof Misses Thousands of File Handles: Linux Kernel Secrets Revealed

1. Origin

One evening the author received a message that a machine showed more than 110,000 file handles in the first column of proc/sys/fs/file-nr, while lsof reported only a few thousand.

Running lsof gave a much smaller number, raising the question of where the missing handles went.

2. File descriptor vs. file handle

Each process has an open‑file table (fdtable). Each entry is a struct file that stores attributes such as offset and access mode; this is the true file handle. The file descriptor is merely an integer index into the fdtable that points to the struct file.

3. What does the first field of file‑nr represent?

Inspecting the kernel source shows that the file‑nr metric is derived from the global variable nr_files, which counts the number of allocated struct file objects, i.e., file handles, not file descriptors.

4. Where are file handles allocated?

The kernel function get_empty_filp in fs/file_table.c allocates a new struct file. The following operations eventually call this function:

open system call (path_openat)

opening a directory (dentry_open)

shared‑memory attach (do_shmat)

socket allocation (sock_alloc_file)

pipe creation (create_pipe_files)

anonymous inode creation for epoll, inotify, signalfd, etc. (anon_inode_getfile)

5. Finding the culprit

Using ipcs the author discovered a shared‑memory segment that had been attached more than 90,000 times, matching the missing handles.

Repeatedly attaching the same shared memory is allowed by the kernel, but it leaves many struct file objects allocated.

6. Why does file‑nr still differ from lsof on a “normal” system?

Even on a typical system, file‑nr and lsof can diverge because lsof reports file descriptors, not file handles. Two experiments illustrate this:

1) A program opens /dev/zero once and duplicates the descriptor 1,000 times with dup. file‑nr changes only slightly, while lsof shows an increase of about 1,000.

2) A program opens /dev/zero 1,000 times, then mmap s each file and closes the descriptors. The descriptor count stays low, but the file‑handle count rises by roughly 1,000 because the kernel’s reference count on the struct file is incremented by the mapping.

These examples demonstrate that many kernel objects have reference counts; closing a descriptor does not necessarily free the underlying struct file.

7. Detecting file handles hidden by memory mappings

Both mmap and shared‑memory attachments allocate memory regions in the process address space. The pmap command can reveal these regions.

On the original machine with 110,000 handles, the slab cache showed a large number of struct file objects as well as a massive amount of vm_area_struct memory.

8. Other cases where lsof misses handles

If a multithreaded service’s main thread exits while worker threads remain alive, the process’s file‑descriptor table may appear empty, causing lsof to miss the handles that are still in use.

9. Summary

Linux kernel metrics such as file‑nr provide valuable insight for system monitoring. Understanding the objects behind these metrics—file handles, shared memory, mmap regions, and thread lifecycles—helps pinpoint the root cause of anomalies when the numbers reported by tools like lsof appear inconsistent.