Understanding Linux Memory Management: From Process Allocation to OOM and Cache

1. Process Memory Allocation

When a program is started, the terminal invokes exec to load the executable; the code, data, BSS, and stack segments are mapped into the process address space via mmap, while the heap is created on demand. After exec, control is handed to the dynamic linker which loads required shared libraries, and the process begins execution. The strace tool can be used to trace these system calls.

On the first malloc, the kernel checks for an existing VMA for the heap. If none exists, an anonymous mmap region is allocated, a VMA structure is added to the process's mm_struct red‑black tree and list, and the user‑space allocator (ptmalloc, tcmalloc, jemalloc, etc.) manages the returned memory. Large allocations may use mmap directly; the memory remains virtual until first accessed, at which point a physical page is allocated.

When free is called, memory obtained via mmap is released with munmap. Memory obtained via the heap allocator is returned to the allocator, which may later release it back to the kernel. All memory is reclaimed when the process exits.

2. Out‑of‑Memory (OOM) Handling

When the system runs out of memory, the OOM killer selects a victim process based on several factors: resident memory size, runtime, priority, user ID, number of child processes, and the /proc/<pid>/oom_adj value. Each process receives an oom_score; the highest score is killed.

The score can be influenced by writing to /proc/<pid>/oom_adj. Values range from –16 (immune) to 15 (most likely to be killed); setting –17 makes a process effectively VIP and prevents OOM termination.

The kernel parameter /proc/sys/vm/overcommit_memory controls allocation policy:

0 – heuristic overcommit (default); large virtual allocations that exceed physical memory may trigger OOM.

1 – always allow overcommit; OOM occurs only when memory is actually exhausted.

2 – strict limit based on swap + RAM * overcommit_ratio (default ratio 50%).

3. Where Does Allocated Memory Reside?

Linux distinguishes between cache memory and ordinary physical memory. The free command shows three zones: DMA, NORMAL, HIGH, and separates cache (buff/cache) from used memory.

3.1 Shared File Mapping

Code and shared library segments are mapped from executable files. Creating a 1 GB file and mapping it shows an increase in buff/cache, proving that these segments reside in the page cache.

3.2 Private File Mapping

Data segments are private file mappings. After dropping caches ( echo 1 > /proc/sys/vm/drop_caches) and mapping a private file, both used and buff/cache grow by 1 GB, illustrating copy‑on‑write behavior.

3.3 Private Anonymous Mapping

Heap, BSS, and anonymous mmap regions are private anonymous mappings. Mapping 1 GB anonymously increases only the used counter; the page cache is untouched.

3.4 Shared Anonymous Mapping

When parent and child processes share memory via mmap with MAP_SHARED, the memory is allocated from the page cache, as shown by a 1 GB increase in buff/cache while used stays unchanged.

4. Memory Reclamation

Two ways to free memory when it is scarce:

Manual reclamation – writing to /proc/sys/vm/drop_caches:

1 – free reclaimable page cache.

2 – free dentries and inodes.

3 – free both.

Dirty pages must be flushed first with sync.

Automatic reclamation – the kernel thread kswapd periodically scans LRU lists, moves inactive pages to the reclaim list, and frees them until the free‑page target is met. If memory pressure is extreme, a more aggressive reclaim pass runs.

File pages are written back if dirty, then freed; clean file pages are simply discarded. Anonymous pages cannot be written back, so they are swapped out to disk and marked for later swap‑in.

The sysctl /proc/sys/vm/vfs_cache_pressure (0‑100) controls the aggressiveness of cache reclamation versus swap usage. Higher values favor swapping, lower values favor keeping cache.

5. Summary

The article revisits the Linux process address space, explains how memory is allocated via mmap, brk, and heap allocators, describes OOM victim selection, shows where different kinds of memory (shared file, private file, anonymous) reside, and outlines both manual ( drop_caches) and automatic (kswapd) reclamation mechanisms.