How Linux Kernel Manages Memory: Allocation, OOM, and Recovery

1. Process Memory Allocation

When a program is started, the terminal calls

exec

to load the executable into memory; the code, data, BSS, and stack segments are mapped via

mmap

, while the heap is created on demand. After

exec

, the dynamic linker loads required shared libraries before the process begins execution, which can be traced with

strace

.

On the first

malloc

, the kernel handles a

brk

system call. If no heap VMA exists, the kernel creates an anonymous mapping with

mmap

and adds the VMA to the process's red‑black tree. The user‑space allocator (ptmalloc, tcmalloc, jemalloc, etc.) then subdivides this region and returns the requested block. Large allocations may bypass the heap and use

mmap

directly; the returned memory is virtual until first accessed, at which point physical pages are allocated.

When

free

is called, memory obtained via

mmap

is released with

munmap

. Memory obtained via the heap is returned to the allocator, which may later give it back to the kernel.

2. OOM After Memory Exhaustion

The OOM (Out‑of‑Memory) killer selects a process to terminate when the system runs out of memory. Selection factors include memory usage, runtime, priority, user ID, number of child processes, and the

oom_adj

score. The kernel computes an

oom_score

for each process; the highest score is killed.

Administrators can influence the decision by writing to

/proc/<pid>/oom_adj

. Values range from –16 (immune) to 15 (most likely to be killed). Setting

oom_adj

to –17 gives a process VIP‑like protection.

The

/proc/sys/vm/overcommit_memory

setting controls allocation policy:

0 – heuristic OOM: modest over‑commit is allowed, but huge virtual allocations trigger OOM.

1 – always allow over‑commit; OOM occurs only when physical memory is truly exhausted.

2 – never exceed

swap + RAM * overcommit_ratio

; allocation fails once the limit is reached.

3. Where Allocated Memory Resides

Linux uses two main mapping types:

File mappings (code, data, shared libraries) are cached in the page cache. When multiple processes map the same file, they share the same physical pages.

Anonymous mappings (heap, BSS, stack,

malloc

via

brk

or

mmap

) are not backed by a file and reside in regular RAM until swapped out.

Experiments show that shared file mappings increase

buff/cache

, while private anonymous mappings increase only

used

memory.

Shared anonymous mappings (e.g.,

mmap

with

MAP_SHARED

) also use the page cache; the memory appears in

buff/cache

and is visible to all participating processes.

Tmpfs (including

/dev/shm

) creates files in a memory‑backed filesystem. These files are stored in the page cache and cannot be reclaimed while they are referenced, but they can be swapped out.

POSIX and System V shared memory are implemented on top of tmpfs, so their pages are also part of the page cache and share the same reclamation constraints.

4. Memory Reclamation

4.1 Manual Reclamation

Writing to

/proc/sys/vm/drop_caches

forces the kernel to drop clean caches:

echo 1 > /proc/sys/vm/drop_caches # drop page cache echo 2 > /proc/sys/vm/drop_caches # drop dentries and inodes echo 3 > /proc/sys/vm/drop_caches # drop both

Dirty pages must be flushed with

sync

before they can be dropped.

4.2 Automatic Reclamation

The kernel’s

kswapd

daemon periodically scans LRU lists. It moves inactive pages to the reclaimable list and frees them until the free‑page target (

pages_high

) is reached. When memory pressure exceeds a critical threshold, a more aggressive reclaim pass runs.

File pages are reclaimed by writing back dirty data and then freeing the cache. Anonymous pages are reclaimed by swapping them out to disk.

The

vm.swappiness

parameter (0‑100) controls the balance between swapping anonymous pages and reclaiming cache; higher values favor swapping.

5. Summary

The article reviewed the Linux process address space, explained how memory is allocated via

brk

and

mmap

, described the OOM killer’s decision process and over‑commit policies, distinguished between file‑backed and anonymous mappings, and covered both manual (

drop_caches

) and automatic (kswapd, swap) memory reclamation mechanisms.