Don’t Change Linux Page Size to 2 MB: How the “Huge Page” Trick Can Crash Your Service

Why Huge Pages Seem Attractive

Linux uses 4 KB standard pages by default. Each page requires a page‑table entry, and the CPU’s Translation Lookaside Buffer (TLB) caches these entries. With a 1 GB allocation, the default layout creates about 262 144 entries, overwhelming the tiny TLB and causing frequent TLB misses that cost dozens of CPU cycles.

Switching to 2 MB huge pages reduces the number of pages by a factor of 512: the same 1 GB needs only 512 entries, allowing the TLB to hold the entire mapping and achieve near‑100 % hit rate. The address‑translation overhead drops dramatically, which can improve raw memory‑access speed.

Benchmark Demonstration

#include <stdio.h>
#include <sys/mman.h>
#include <string.h>
#include <sys/time.h>
#define SIZE (1UL << 30)   // 1 GB
long long get_time_us(){
    struct timeval tv;
    gettimeofday(&tv, NULL);
    return tv.tv_sec*1000000LL + tv.tv_usec;
}
int main(){
    // 4 KB pages
    char* normal_mem = mmap(NULL, SIZE, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
    // 2 MB huge pages
    char* huge_mem = mmap(NULL, SIZE, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0);
    long long start = get_time_us();
    for (long i=0; i<SIZE; i+=4096) normal_mem[i]=1;
    long long time_normal = get_time_us() - start;
    start = get_time_us();
    for (long i=0; i<SIZE; i+=2048*1024) huge_mem[i]=1;
    long long time_huge = get_time_us() - start;
    printf("4KB page traversal: %lld µs
", time_normal);
    printf("2MB huge page traversal: %lld µs
", time_huge);
    return 0;
}

Running this on a machine with huge pages enabled shows the 2 MB version completing in a fraction of the time of the 4 KB version, confirming the TLB‑hit advantage.

Real‑World Microservice Failure

The author applied the same configuration change to a typical microservice cluster, disabling Transparent Huge Pages (THP) and allocating static 2 MB huge pages for the whole system. Initial monitoring showed a sharp drop in TLB‑miss counters, leading to the false impression that the optimization succeeded.

Within minutes, the services began to time out and crash. The root cause was examined step by step:

Each microservice request allocates only about 1 KB of memory.

With 4 KB pages, the kernel allocates a single 4 KB page per request, keeping memory usage low.

With 2 MB huge pages, the kernel must allocate a whole 2 MB page for each 1 KB request, causing rapid exhaustion of the pre‑reserved huge‑page pool and a 30 %+ jump in overall memory consumption.

The allocated huge pages cannot be reclaimed or swapped, so once the pool is depleted new allocations fail, leading to service errors and cascading failures.

Additional side effects observed:

Cache‑coherency traffic (MESI protocol) surged because many threads accessed data within the same huge page, increasing lock contention and soft‑interrupt counts.

Page‑fault latency grew: handling a 2 MB page fault is far more expensive than a 4 KB fault, inflating tail‑latency (P99/P999) and causing noticeable response‑time spikes.

The kernel’s khugepaged thread continued to scan and attempt to merge pages, adding further CPU overhead and periodic performance jitter.

Code Illustrating the Allocation Pattern

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
// Each request allocates 1 KB
#define REQ_MEM_SIZE 1024
int main(){
    int alloc_cnt = 0;
    while (1){
        void *buf = malloc(REQ_MEM_SIZE);
        if (buf == NULL){
            printf("Memory allocation failed – huge pages exhausted, service abnormal
");
            break;
        }
        alloc_cnt++;
        printf("Allocation %d: 1KB request
", alloc_cnt);
        usleep(100);
        free(buf);
    }
    return 0;
}

Running this program on a system using 4 KB pages proceeds indefinitely, while on a system forced to use 2 MB huge pages it quickly exhausts memory and aborts, mirroring the production outage.

Rollback Procedure

To recover, the page‑size configuration was reverted:

# Disable huge pages
vm.nr_hugepages = 0
# Comment out the explicit huge‑page size (defaults to 4 KB)
# vm.hugepagesz = 2MB

After applying the changes with sudo sysctl -p, unmounting /dev/hugepages, and rebooting, the system returned to its original performance: average response time fell back to ~550 ms, throughput rose above 900 req/s, and memory usage stabilized.

Takeaways

The key lesson is that performance parameters are trade‑offs. Huge pages excel for workloads that continuously occupy large, contiguous memory regions (databases, caches, DPDK, HPC). For typical microservices with many tiny, short‑lived allocations, the default 4 KB pages provide better memory efficiency, lower cache‑conflict rates, and more predictable latency. Optimizations must be evaluated against the specific access pattern and resource constraints of the target application.