Why 8 GB Memory Allocation Fails on 4 GB Linux Machines and How Overcommit & Swap Change the Outcome

Background and Problem

A reader reported that on a 64‑bit Linux system with 2 GB physical RAM, requesting 8 GB of memory sometimes fails, prompting an investigation into the role of the overcommit_memory kernel parameter.

Three Preconditions for the 8 GB Question

Is the operating system 32‑bit or 64‑bit?

Will the allocated 8 GB be actually used?

Does the OS have swap enabled?

Virtual Memory Basics

When a program calls malloc, it reserves virtual memory without immediately allocating physical RAM. Accessing that memory triggers a page‑fault; the kernel then either maps a free physical page or, if none are available, starts reclaiming memory and may invoke the OOM (Out‑of‑Memory) killer.

In Linux, the virtual address space is split into kernel space and user space. On a 32‑bit system the user space is 3 GB (kernel space 1 GB). On a 64‑bit system each space can be up to 128 TB.

32‑bit Scenario

On a 32‑bit OS with 4 GB physical RAM, an 8 GB allocation fails during the virtual‑memory reservation stage because the maximum user‑space size is only 3 GB.

64‑bit Scenario

On a 64‑bit OS, a process can request up to 128 TB of virtual memory, so allocating 8 GB succeeds as long as the memory is not accessed.

Experiment 1: Allocating and Accessing 4 GB on a 2 GB Machine (No Swap)

The following C program allocates four 1 GB blocks and then accesses each block with memset:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#define MEM_SIZE 1024*1024*1024
int main() {
    char* addr[4];
    for (int i=0;i<4;++i) {
        addr[i] = (char*)malloc(MEM_SIZE);
        if (!addr[i]) { printf("malloc failed: %s
", strerror(errno)); return -1; }
        printf("Allocated block %d at %p
", i, addr[i]);
    }
    for (int i=0;i<4;++i) {
        printf("Touching block %d
", i+1);
        memset(addr[i], 0, MEM_SIZE);
    }
    getchar();
    return 0;
}

Running this on a 64‑bit server with 2 GB RAM (no swap) shows that the process is killed after the second block is touched, and the kernel logs an OOM event.

Understanding OOM

OOM (Out‑of‑Memory) occurs when the kernel cannot satisfy a memory request even after reclaiming pages, leading to the termination of the highest‑scoring process.

The overcommit_memory Parameter

Linux provides three modes:

0 (default) – heuristic overcommit; large single allocations may be rejected.

1 – always allow overcommit.

2 – never overcommit.

When the parameter is 0, a request for 4 GB on a 2 GB machine can be denied with “Cannot allocate memory”. Setting it to 1 with echo 1 > /proc/sys/vm/overcommit_memory allows the allocation to succeed.

echo 1 > /proc/sys/vm/overcommit_memory

Swap Mechanism

Swap extends usable memory by moving rarely used pages to disk. Without swap, exceeding physical RAM triggers immediate OOM. With swap, the kernel can page out idle pages, allowing larger virtual memory usage at the cost of disk I/O.

Experiment 2: Accessing 32 GB with Swap Enabled

On a macOS laptop (64‑bit, 8 GB RAM, 1 GB swap), the following program allocates 32 GB of virtual memory and repeatedly writes to it:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define MEM_SIZE 32*1024*1024*1024
int main() {
    char* addr = (char*)malloc((long)MEM_SIZE);
    printf("Allocated 32 GB virtual memory
");
    while (1) {
        printf("Touching 32 GB...
");
        memset(addr, 0, (long)MEM_SIZE);
    }
    return 0;
}

The program runs continuously; the system shows ~2.3 GB of swap usage and high disk I/O, confirming that swap allows the process to use memory far beyond the physical limit.

Limits of Swap

Even with swap, the kernel eventually runs out of reclaimable pages. When a program requests 64 GB and accesses about 56 GB, the system kills the process after repeated reclamation attempts, demonstrating that swap is not unlimited.

Summary

On 32‑bit Linux, the maximum user‑space is ~3 GB; an 8 GB allocation fails at the virtual‑memory reservation stage.

On 64‑bit Linux, up to 128 TB of virtual memory can be reserved; allocation succeeds as long as the memory is not touched.

If the memory is accessed, the outcome depends on swap:

No swap → OOM killer terminates the process.

Swap enabled → the kernel pages out idle pages, allowing the process to use memory far beyond physical RAM, though performance degrades due to disk I/O.

The overcommit_memory setting controls whether large allocations are allowed; setting it to 1 disables the heuristic check.