How Linux Manages Process Memory: From Allocation to OOM Recovery

1. Process Memory Allocation and Mapping

When a program is started from a terminal, the shell calls exec to load the executable into memory. The code, data, BSS, and stack segments are mapped via mmap, while the heap is created either by extending the program break with brk or by anonymous mmap when large allocations are requested.

After exec, control is handed to the dynamic linker, which loads required shared libraries. The whole sequence can be observed with strace. The first call to malloc triggers a brk system call; if no VMA for the heap exists, the kernel creates one with an anonymous mapping and adds it to the process's mm_struct red‑black tree.

Memory allocators such as ptmalloc, tcmalloc, or jemalloc manage the returned region and hand it to the user. For very large allocations, malloc may call mmap directly, returning virtual memory that is backed by physical pages only upon first access.

When free releases memory, if the region was obtained via mmap the kernel unmaps it with munmap. Otherwise the memory is returned to the allocator, which may later give it back to the kernel. All memory is reclaimed when the process exits.

2. Out‑of‑Memory (OOM) Handling

When the system runs out of memory, the OOM killer selects a victim process to terminate. The selection is based on an oom_score calculated from factors such as resident memory size, runtime, priority, user ID, number of child processes, and the oom_adj value.

Administrators can influence the decision by writing a score to /proc/<pid>/oom_adj. Values range from –16 (immune) to 15 (most likely to be killed). Setting it to –17 makes the process effectively VIP‑protected.

The kernel parameter /proc/sys/vm/overcommit_memory controls allocation policy:

0 – heuristic overcommit: allocations are allowed unless the virtual request vastly exceeds physical memory.

1 – always allow overcommit, relying on OOM only when memory truly runs out.

2 – strict limit based on swap + RAM * overcommit_ratio, rejecting allocations once the limit is reached.

3. Where Allocated Memory Resides

Linux distinguishes between cache memory and ordinary physical memory. Memory obtained through shared file mappings (code and shared libraries) is first read into the page cache and then mapped into each process's address space.

Private file mappings also use the page cache; modifications trigger a copy‑on‑write, allocating new private pages.

Anonymous private mappings (heap, BSS, stack) are allocated directly as physical pages without involving the cache.

Shared anonymous mappings (used for inter‑process communication) are backed by the page cache, allowing multiple processes to see the same physical pages.

4. System Memory Reclamation

Two manual ways exist to free memory:

Writing 1, 2, or 3 to /proc/sys/vm/drop_caches releases page cache, dentries, and inodes respectively.

Flushing dirty pages with sync before dropping caches.

Tmpfs, procfs, sysfs, and ramfs are memory‑based filesystems. Files stored in tmpfs reside in the page cache and cannot be reclaimed while they are referenced, though they may be swapped out.

POSIX and System V shared memory are implemented on top of tmpfs; their pages are also non‑reclaimable until the shared segment is explicitly removed.

Automatic reclamation is performed by the kernel thread kswapd, which periodically scans LRU lists and moves inactive pages to free lists. File pages are written back if dirty before being freed; anonymous pages are swapped out.

The vm.swappiness parameter (0–100) tunes the preference for swapping versus cache reclamation.

5. Summary

The article reviewed the Linux process address space, described how memory is allocated via brk and mmap, explained OOM killer logic and overcommit settings, demonstrated where different mappings reside (cache vs ordinary memory), and covered both manual and automatic memory reclamation mechanisms.