Why Does Linux Need Complex Memory Management? Inside Virtual Memory, malloc, and OOM

Virtual Memory Mechanism

Linux isolates each process in its own virtual address space and maps virtual addresses to physical memory. This indirection solves three fundamental problems:

Process isolation prevents one process from reading or writing another's memory, avoiding crashes and security breaches.

Limited physical memory and dynamic process creation would otherwise cause allocation failures, idle processes holding memory, and fragmentation.

Uniform virtual addresses simplify compiler, linker and debugger design.

The kernel’s memory manager implements the following functions:

Assign an independent virtual address space to every process.

Maintain page‑table mappings from virtual pages to physical frames.

Perform lazy allocation via page‑fault handling.

Swap out inactive pages to disk.

Invoke the OOM killer when memory is exhausted.

Address Translation

The CPU issues virtual addresses; the Memory Management Unit (MMU) translates them to physical addresses using the process’s page tables. To avoid a single massive table, Linux uses multi‑level page tables (e.g., two‑level on 32‑bit systems). A 4 GB virtual space with 4 KB pages would need 2²⁰ entries in a flat table; multi‑level tables reduce memory consumption at the cost of additional lookup steps.

MMU and TLB

The MMU caches recent page‑table entries in a Translation Lookaside Buffer (TLB). On a TLB miss the MMU walks the full page‑table hierarchy and then updates the TLB, dramatically reducing average translation latency.

Page Fault Types

Hard (major) page fault: No physical frame exists; the kernel reads the page from disk (swap or file).

Soft (minor) page fault: A frame exists (e.g., shared memory) but the process lacks a mapping; the kernel only creates the mapping.

Invalid page fault: The process accessed an illegal address (null pointer, out‑of‑bounds).

Typical causes are out‑of‑bounds accesses, delayed allocation via malloc, and pages that have been swapped out.

Memory Allocation

Within each virtual address space the kernel defines several logical segments (illustrated for a 32‑bit system):

text : executable code.

data : initialized globals.

bss : uninitialized globals.

heap : dynamic allocations; starts at start_brk and grows to the program break ( brk).

mmap area : anonymous or file‑mapped regions created by mmap.

stack : call stack, limited by RLIMIT_STACK (commonly 8 MiB).

These regions are not required to be contiguous; random offsets improve security (ASLR).

Kernel Data Structures

Each virtual region is represented by struct vm_area_struct, linked in a doubly‑linked list and also stored in a red‑black tree for fast lookup. The process descriptor struct task_struct contains a pointer to struct mm_struct *mm, which holds: pgd – pointer to the first‑level page table. mmap – head of the vm_area_struct list. mm_rb – root of the red‑black tree of vm_area_struct nodes.

User‑Space Allocators

glibc’s malloc sits on top of kernel allocators (buddy system, slab). It ultimately calls:

#include <unistd.h>
int brk(void *addr);
void *sbrk(intptr_t increment);

For small requests malloc expands the heap via brk / sbrk; for large requests it uses mmap to obtain anonymous memory.

#include <sys/mman.h>
void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
int munmap(void *addr, size_t length);

glibc malloc Internals

Memory is divided into chunks . Free chunks are grouped into bins :

fast bins – 10 size classes (16‑88 bytes) for quick reuse.

unsorted bin – holds recently freed chunks before they are sorted.

small bins (indices 2‑63) – 62 bins for < 512 B chunks, step size 8 bytes.

large bins (indices 64‑126) – 63 bins for ≥ 512 B chunks, sorted by size.

Special chunks:

top chunk – the remaining heap space; used when no bin can satisfy a request.

last remainder chunk – leftover from a previous unsorted‑bin allocation.

#define MAX_FAST_SIZE 80
#define NFASTBINS (fastbin_index(request2size(MAX_FAST_SIZE))+1)

Allocation Flow

Round the requested size to a chunk_size.

If chunk_size ≤ MAX_FAST_SIZE, search fast bins.

If ≤ 512 B, search small bins.

If not found, coalesce adjacent fast‑bin chunks into the unsorted bin.

Attempt to split a suitable unsorted‑bin chunk (or move it to small/large bins).

If still unsatisfied, search large bins using a “smallest‑first, best‑fit” strategy.

Fallback to the top chunk; if insufficient, extend the heap via brk or allocate a new region with mmap.

Fast and small bins require an exact size match; large bins allow a best‑fit search, which can increase fragmentation.

Memory Reclamation

Linux reclaims two categories of pages:

File pages (page cache): reclaimed using reverse mapping and LRU lists (active vs. inactive).

Anonymous pages (heap, stack, private mappings): reclaimed by swapping them out to disk.

kswapd and Watermarks

Each NUMA node runs a kswapd kernel thread. The thread wakes when memory pressure crosses one of three watermarks:

high – ample free memory; kswapd sleeps.

between high and low – moderate pressure; kswapd reclaims pages.

low to min – severe pressure; aggressive reclamation.

below min – immediate reclamation.

Out‑of‑Memory (OOM) Killer

When memory is critically low, the OOM killer selects a victim process based on a score that combines RSS, swap usage, and configurable adjustments ( OOM_SCORE_ADJ). The selected process is terminated to free memory.

// Example scoring snippet from Linux source
// https://github.com/torvalds/linux/blob/master/mm/oom_kill.c

Conclusion

This summary covered why Linux needs virtual memory, the core mechanisms (page tables, MMU/TLB, page‑fault handling), the layout of a process’s address space, glibc’s malloc data structures and allocation algorithm, and the memory‑reclamation subsystem (kswapd watermarks and OOM killer).