Understanding Linux OOM Killer: How It Works and How to Prevent It

When a Linux system runs out of physical memory, the kernel first tries to reclaim memory from page cache and unused slab pages. If these are insufficient, the OOM (Out‑Of‑Memory) mechanism triggers the OOM killer, which selects the process consuming the most memory and terminates it to keep the system alive.

What is the OOM Mechanism?

OOM stands for Out Of Memory. The kernel’s emergency response activates when physical memory is exhausted. It first frees reclaimable memory such as file page cache and delayed‑release slab pages, which do not affect normal operation.

If memory pressure persists, the kernel invokes the OOM killer to kill the process with the highest memory usage.

Implementation of the OOM Killer

The kernel implements OOM killer through several functions: pagefault_out_of_memory() calls out_of_memory() when a page fault cannot allocate physical memory. out_of_memory() selects a bad process via select_bad_process() and then calls oom_kill_process() to terminate it.

void pagefault_out_of_memory(void) {
    ...
    out_of_memory(NULL, 0, 0, NULL, false);
    ...
}

void out_of_memory(struct zonelist *zonelist, gfp_t gfp_mask, int order,
                   nodemask_t *nodemask, bool force_kill) {
    ...
    p = select_bad_process(&points, totalpages, mpol_mask, force_kill);
    if (p != (void *)-1U) {
        oom_kill_process(p, gfp_mask, order, points, totalpages, NULL,
                         nodemask, "Out of memory");
        killed = 1;
    }
    ...
}

The core logic of out_of_memory() is to find the worst‑scoring process and kill it.

How the Kernel Chooses the Victim

select_bad_process()

iterates over all processes and threads, computes a score using oom_badness(), and picks the one with the highest score.

static struct task_struct *select_bad_process(unsigned int *ppoints,
                     unsigned long totalpages, const nodemask_t *nodemask,
                     bool force_kill) {
    struct task_struct *g, *p, *chosen = NULL;
    unsigned long chosen_points = 0;
    ...
    for_each_process_thread(g, p) {
        unsigned int points;
        points = oom_badness(p, NULL, nodemask, totalpages);
        if (!points || points < chosen_points)
            continue;
        chosen = p;
        chosen_points = points;
    }
    return chosen;
}

The scoring function oom_badness() follows four steps:

If the process is unkillable (e.g., init or kernel threads), return 0.

Read the process’s oom_score_adj value from /proc/<pid>/oom_score_adj; lower values reduce kill probability.

Count physical memory used: RSS, page tables, and swap entries.

Add the adjusted score to the memory usage to obtain the final points.

unsigned long oom_badness(struct task_struct *p, struct mem_cgroup *memcg,
            const nodemask_t *nodemask, unsigned long totalpages) {
    long points, adj;
    if (oom_unkillable_task(p, memcg, nodemask))
        return 0;
    adj = (long)p->signal->oom_score_adj;
    if (adj == OOM_SCORE_ADJ_MIN)
        return 0;
    points = get_mm_rss(p->mm) + atomic_long_read(&p->mm->nr_ptes) +
             get_mm_counter(p->mm, MM_SWAPENTS);
    adj *= totalpages / 1000;
    points += adj;
    return points > 0 ? points : 1;
}

Preventing a Process from Being Killed

Critical services (e.g., MySQL) can be protected by setting oom_score_adj to -1000, which makes the kernel treat the process as unkillable. $ echo -1000 > /proc/2000/oom_score_adj After this adjustment, the OOM killer will skip the specified process, reducing the risk of data loss.

Overall, understanding the OOM killer’s workflow—memory reclamation, victim selection, and scoring—allows administrators to tune the system and safeguard essential applications.