How Linux Manages Memory: From Process Allocation to OOM and Cache

During an internship the author became interested in Linux kernel memory management after an OOM talk and decided to document the findings.

Introduction

The article analyzes a single process's memory layout and allocation from a global perspective of kernel memory management.

Process memory request and allocation

OOM after memory exhaustion

Where the allocated memory resides

System memory reclamation

1. Process Memory Allocation

When a program is started, the terminal calls exec to load the executable into memory, mapping the code, data, BSS, and stack segments via mmap. The heap is mapped when memory is requested.

After exec, control is handed to the dynamic linker, which loads required shared libraries before the process begins execution. This can be observed with strace.

On the first malloc, the kernel checks for a heap VMA; if none exists, it creates an anonymous mapping and adds a VMA to the mm_struct red‑black tree.

Memory allocated via brk is virtual until first access, at which point it is backed by physical memory. Large allocations may use mmap directly.

When free is called, memory obtained via mmap is released with munmap; otherwise it is returned to the allocator, which later returns it to the system.

2. OOM After Memory Exhaustion

When the system runs out of memory, the OOM killer selects a process to kill based on factors such as memory usage, runtime, priority, user, child count, and oom_adj value.

The kernel computes an oom_score for each process; the highest score is chosen for termination. The /proc/<pid>/oom_adj file can be used to influence this score.

The /proc/sys/vm/overcommit_memory setting controls allocation behavior:

0 – heuristic overcommit (default)

1 – always allow overcommit

2 – never exceed a limit based on swap + RAM

3. Where Does Allocated Memory Reside?

3.1 Shared File Mapping

Code and shared library segments are mapped from the executable file and reside in the kernel page cache. Experiments with a 1 GB file show buff/cache increasing by about 1 GB.

3.2 Private File Mapping

Data segments are privately mapped. When a private file mapping is modified, the kernel copies the data to a new private page (copy‑on‑write), increasing both used and buff/cache by the file size.

3.3 Private Anonymous Mapping

Segments such as BSS, heap, and stack are anonymous mappings. They increase only used memory without affecting buff/cache.

3.4 Shared Anonymous Mapping

Shared anonymous mappings (e.g., between parent and child after fork) allocate memory from the page cache, increasing buff/cache while the underlying pages are shared.

4. Memory Reclamation

4.1 Manual Reclamation

Writing 1 to /proc/sys/vm/drop_caches releases reclaimable page cache; 2 releases dentries and inodes; 3 releases both.

Dirty pages must be flushed with sync before they can be dropped.

4.2 tmpfs

tmpfs is an in‑memory filesystem that uses the page cache and can also swap out. Files created in /dev/shm increase buff/cache by their size.

4.3 Shared Memory

POSIX and System V shared memory are implemented on top of tmpfs, creating a file in the tmpfs and mapping it into processes. These pages are not reclaimable while the shared memory segment exists.

4.4 Automatic Memory Release

The kernel’s kswapd daemon periodically scans LRU lists and reclaims pages. When memory pressure is high, a more aggressive reclamation is triggered.

File pages are reclaimed by writing back dirty pages or simply dropping clean pages. Anonymous pages are swapped out because they have no backing store.

The vm.swappiness parameter (0‑100) controls the tendency to swap anonymous pages versus reclaiming cache.

5. Summary

The article reviewed process address space, explained how Linux handles memory exhaustion via manual cache dropping or automatic reclamation by kswapd, and described the OOM killer’s decision process. It highlighted that the kernel works hard to free memory through cache management, swapping, and, when necessary, killing processes.