Unlocking Linux Memory Management: From Process Allocation to OOM Handling

Preface

During an internship the author became fascinated by Linux kernel memory management and, after accumulating knowledge, writes this article to share insights.

Overview of Linux Memory Management

The article analyses a single process's memory layout and allocation from a global perspective, covering process memory requests, OOM handling, the location of allocated memory, and system memory reclamation.

1. Process Memory Requests and Allocation

When a program is started, exec loads the executable, mapping the code, data, BSS, and stack segments via mmap. The heap is created on demand using brk or anonymous mmap. The kernel builds VMA structures and links them into the mm_struct.

Memory allocated with malloc first checks for an existing heap VMA; if none, an anonymous mapping is created. Large allocations may use mmap directly. The allocator (ptmalloc, tcmalloc, jemalloc) manages the returned memory. When free is called, memory obtained via mmap is released with munmap; otherwise it is returned to the allocator.

2. OOM (Out‑of‑Memory) Handling

When the system runs out of memory the OOM killer selects a process to terminate. Selection is based on oom_score, which considers memory usage, runtime, priority, user, number of child processes, and the oom_adj parameter. Adjusting /proc/<pid>/oom_adj can protect a process (e.g., setting it to –17).

The kernel parameter /proc/sys/vm/overcommit_memory controls allocation behavior: 0 = heuristic overcommit, 1 = always allow, 2 = never exceed a limit based on swap + RAM × ratio.

3. Where Does Allocated Memory Reside?

3.1 Shared File Mapping

Code and shared libraries are mapped from the executable file. They are first read into the page cache and then mapped into each process’s address space.

3.2 Private File Mapping

Data segments are privately mapped. The file is cached, and on write a copy‑on‑write allocation creates a separate physical page for the modifying process.

3.3 Private Anonymous Mapping

Segments such as .bss, heap, and stack are anonymous and private; they consume physical memory directly without involving the page cache.

3.4 Shared Anonymous Mapping

When parent and child share memory via mmap with MAP_SHARED, the pages are allocated from the cache and mapped into both processes.

4. Memory Reclamation

4.1 Manual Reclamation

Writing to /proc/sys/vm/drop_caches (1 = page cache, 2 = dentries/inodes, 3 = both) frees reclaimable cache pages. Dirty pages must be flushed with sync first.

echo 1 >> /proc/sys/vm/drop_caches

4.2 tmpfs

tmpfs, like ramfs, stores files in memory (and optionally swap). Files created in tmpfs occupy page‑cache memory that cannot be dropped because they are referenced.

4.3 Shared Memory (POSIX & System V)

Both mechanisms create a file in tmpfs and map it, so the memory resides in the cache and is not reclaimable by drop_caches.

4.4 Automatic Reclamation

The kernel’s kswapd daemon scans LRU lists, moves inactive pages to the reclaim list, and frees them either by writing dirty file pages back to disk or swapping out anonymous pages. The vm.swappiness parameter controls the preference for swapping versus cache reclamation.

Conclusion

The article reviewed the process address space, described how Linux allocates and reclaims memory, explained OOM selection criteria, and highlighted the role of page cache, tmpfs, and swap in memory management.