Why lsof Misses Thousands of File Handles: Linux Kernel Secrets Revealed

1. Origin

One evening the author received a message that a machine showed more than 110,000 file handles in the first column of

proc/sys/fs/file-nr

, while

lsof

reported only a few thousand.

Running

lsof

gave a much smaller number, raising the question of where the missing handles went.

2. File descriptor vs. file handle

Each process has an open‑file table (fdtable). Each entry is a

struct file

that stores attributes such as offset and access mode; this is the true file handle. The file descriptor is merely an integer index into the fdtable that points to the

struct file

.

3. What does the first field of file‑nr represent?

Inspecting the kernel source shows that the

file‑nr

metric is derived from the global variable

nr_files

, which counts the number of allocated

struct file

objects, i.e., file handles, not file descriptors.

4. Where are file handles allocated?

The kernel function

get_empty_filp

in

fs/file_table.c

allocates a new

struct file

. The following operations eventually call this function:

open system call (path_openat)

opening a directory (dentry_open)

shared‑memory attach (do_shmat)

socket allocation (sock_alloc_file)

pipe creation (create_pipe_files)

anonymous inode creation for epoll, inotify, signalfd, etc. (anon_inode_getfile)

5. Finding the culprit

Using

ipcs

the author discovered a shared‑memory segment that had been attached more than 90,000 times, matching the missing handles.

Repeatedly attaching the same shared memory is allowed by the kernel, but it leaves many

struct file

objects allocated.

6. Why does file‑nr still differ from lsof on a “normal” system?

Even on a typical system,

file‑nr

and

lsof

can diverge because

lsof

reports file descriptors, not file handles. Two experiments illustrate this:

1) A program opens

/dev/zero

once and duplicates the descriptor 1,000 times with

dup

.

file‑nr

changes only slightly, while

lsof

shows an increase of about 1,000.

2) A program opens

/dev/zero

1,000 times, then

mmap

s each file and closes the descriptors. The descriptor count stays low, but the file‑handle count rises by roughly 1,000 because the kernel’s reference count on the

struct file

is incremented by the mapping.

These examples demonstrate that many kernel objects have reference counts; closing a descriptor does not necessarily free the underlying

struct file

.

7. Detecting file handles hidden by memory mappings

Both

mmap

and shared‑memory attachments allocate memory regions in the process address space. The

pmap

command can reveal these regions.

On the original machine with 110,000 handles, the slab cache showed a large number of

struct file

objects as well as a massive amount of

vm_area_struct

memory.

8. Other cases where lsof misses handles

If a multithreaded service’s main thread exits while worker threads remain alive, the process’s file‑descriptor table may appear empty, causing

lsof

to miss the handles that are still in use.

9. Summary

Linux kernel metrics such as

file‑nr

provide valuable insight for system monitoring. Understanding the objects behind these metrics—file handles, shared memory, mmap regions, and thread lifecycles—helps pinpoint the root cause of anomalies when the numbers reported by tools like

lsof

appear inconsistent.