Understanding Linux Kernel Memory Management: From Process Allocation to OOM

1. Process Memory Allocation

When a program is started, the terminal calls exec to load the executable. The code, data, BSS and stack segments are mapped into the process address space via mmap, while the heap appears only after a malloc request.

During the first malloc, the kernel checks for an existing VMA for the heap; if none exists it creates an anonymous mapping with mmap and links the VMA into the process's mm_struct. The memory allocator (ptmalloc, tcmalloc, jemalloc) then slices the region and returns the requested block.

Large allocations may bypass the heap and use mmap directly; the returned region stays virtual until the first access triggers a real page allocation. brk also returns virtual memory that becomes physical after the allocator touches it.

When free is called, memory obtained via mmap is released with munmap. Heap memory is returned to the allocator, which may later give it back to the kernel, which is why accessing freed memory does not always fault immediately. All memory is reclaimed when the process exits.

2. Out‑of‑Memory (OOM) Handling

When the system runs out of memory, the OOM killer selects a process to terminate. Selection is based on several factors: memory consumption, runtime, priority, user ID, number of child processes, and the oom_adj score. The process with the highest oom_score is killed.

Administrators can influence the decision by writing a value to /proc/<pid>/oom_adj. Values range from –16 (immune) to 15 (most likely to be killed). The kernel parameter /proc/sys/vm/overcommit_memory controls allocation policies:

0 – heuristic overcommit (default).

1 – always allow overcommit.

2 – strict limit based on swap + RAM * overcommit_ratio.

3. Where Allocated Memory Resides

3.1 Shared File Mappings

Code and shared libraries are mapped from the executable file. They are cached in the page cache; multiple processes share the same physical pages.

3.2 Private File Mappings

Data segments that must be private are mapped as private file mappings. When modified, the kernel copies the pages (copy‑on‑write) to a new private area, increasing both used and buff/cache.

3.3 Private Anonymous Mappings

Segments such as BSS, heap, and stack are anonymous private mappings. They consume physical memory directly without involving the page cache.

3.4 Shared Anonymous Mappings

When parent and child processes share memory via mmap with MAP_SHARED, the pages are placed in the cache and mapped into each process's virtual address space.

3.5 tmpfs and Page Cache

tmpfs, procfs, sysfs and ramfs are memory‑based filesystems. Files created in tmpfs are stored in the page cache and may also use swap. The kernel reads and writes these files through the cache, so they appear as cache usage in free.

4. Memory Reclamation

4.1 Manual Reclamation

Writing 1, 2 or 3 to /proc/sys/vm/drop_caches frees page cache, dentries and inodes respectively. Dirty pages must be flushed with sync before they can be dropped.

echo 1 >> /proc/sys/vm/drop_caches

4.2 Automatic Reclamation

The kernel thread kswapd periodically checks free memory. If it falls below pages_low, it scans the LRU lists, moves inactive pages to the inactive list, and frees them until pages_high is reached. When memory is critically low, a more aggressive reclaim is performed.

File pages are reclaimed by writing them back to disk (if dirty) or simply discarding them (if clean). Anonymous pages are swapped out because they have no backing store.

The sysctl vm.swappiness (0‑100) controls the balance between reclaiming cache and swapping anonymous pages.

5. Summary

The article reviews the Linux process address space, explains how memory is allocated and freed, describes OOM selection criteria, shows where different kinds of memory reside (cache vs. used), and outlines both manual and automatic memory reclamation techniques.