Hidden Slab Memory: Why ‘free’ Shows Missing GB on CentOS and How to Fix It

Hidden Slab Memory: Why ‘free’ Shows Missing GB on CentOS and How to Fix It

While maintaining a CentOS server I noticed that the memory reported by free was far lower than expected, with a discrepancy of several gigabytes compared to the sum of process memory shown by ps. After investigation I discovered that the free output does not account for the Slab cache, which can hide large memory usage.

Problem Description and Initial Investigation

The server has 16 GB of RAM, but free -g reported almost all of it as used:

$ free -g
            total       used       free     shared    buffers     cached
Mem:            15         15          0          0          2          0
-/+ buffers/cache:         12          2
Swap:           17          0         17

The table below explains the columns shown by free:

/

total

used

free

shared

buffers

cached

Mem

Total physical memory

Memory in use (including slab, buffers, cached)

Completely unused memory

Shared memory between processes

Metadata of cached files

Actual cached file contents

-/+ buffers/cache

Memory in use (excluding buffers+cached, but including slab)

Free + buffers + cached

Swap

Total swap space

Used swap

Free swap

Running ps aux | awk '{mem += $6} END {print mem/1024/1024}' showed that all processes together used less than 1 GB:

$ ps aux | awk '{mem += $6} END {print mem/1024/1024}'
0.595089

The mismatch is caused by free not counting the Slab cache.

Slab Overview and Further Investigation

Slab Allocation, introduced after Linux 2.2, is a kernel memory management mechanism that caches kernel objects, acting as an object pool to improve performance and reduce fragmentation. Since Linux 2.6.23 the default allocator is SLUB.

Inspecting /proc/meminfo reveals the Slab usage: $ cat /proc/meminfo Relevant lines:

Slab:             154212 kB
SReclaimable:      87980 kB
SUnreclaim:        66232 kB

On the problematic server the Slab size was about 12 GB:

$ cat /proc/meminfo | grep Slab
Slab:         12777668 kB

Because free includes Slab in the “used” column, the apparent memory loss was actually hidden Slab consumption. Detailed Slab statistics can be viewed via /proc/slabinfo or the slabtop command, which showed that ext3_inode_cache and dentry_cache occupied most of the memory.

Solution

If the large Slab is mostly SReclaimable, it is harmless; however, excessive SUnreclaim may indicate a kernel bug. To manually drop reclaimable caches you can write to /proc/sys/vm/drop_caches: echo 2 > /proc/sys/vm/drop_caches After running the command, a subsequent free -g shows a sudden increase of free memory by many gigabytes. Manual cache dropping is temporary and can degrade performance, so it is not recommended for regular use.

The underlying cause was a process (in this case rsync) that caused the Slab to grow; clearing the cache only provides a short‑term relief.

Adjusting System vm Settings

Warning: Changing the following sysctl parameters can negatively affect system performance; test carefully before applying in production.

Edit /etc/sysctl.conf (see vm.txt for documentation).

vm.vfs_cache_pressure

This controls how aggressively the kernel reclaims the page cache, inode cache, dentry cache, and swap cache. Higher values increase the reclaim rate for inode and dentry caches.

Default is 100; values >100 speed up reclamation, values <100 slow it down, and 0 disables reclamation (risking OOM).

vm.min_free_kbytes

Defines the amount of memory reserved for low‑memory situations (e.g., atomic allocations). A larger value triggers earlier memory reclamation, while a too‑large setting can cause OOM, and a too‑small value may lead to deadlocks.

vm.swappiness

Controls the tendency of the kernel to swap out memory. The range is 0–100; higher values increase swapping aggressiveness. The default is 60; setting it to 0 disables swapping.

References

man proc

The Linux Kernel's VFS Layer

The VFS in Linux Kernel V2.4

openSUSE: System Analysis and Tuning Guide, Chapter 15

Red Hat Enterprise Linux, 5.5 Tuning Virtual Memory

Odd behavior

Wikipedia: Slab allocation

Further Reading

Linux System IO Monitoring

Paging

Understanding the Linux Virtual Memory Manager

Understanding the Linux Kernel, 3rd Edition