Why Switching Linux Page Size from 4KB to 2MB Crashed My Production Services

The article opens with a blunt conclusion: 2 MB huge pages are not a universal performance silver bullet; blindly replacing the default 4 KB pages will likely degrade performance instead of improving it.

Why Huge Pages Are Supposed to Help

The core benefit of huge pages is a drastic reduction in address-translation overhead. CPUs translate virtual to physical addresses via the MMU, and the speed of that translation depends on the TLB (Translation Lookaside Buffer), a tiny, ultra-fast hardware cache. With 4 KB pages, 1 GB of memory requires 262,144 page-table entries — far more than the TLB can hold — causing frequent TLB misses. Each miss forces the CPU to walk the four-level page table in main memory, consuming dozens of extra clock cycles.

Switching to 2 MB pages expands each page 512×, so the same 1 GB needs only 512 entries. Those fit easily in the TLB, pushing the hit rate near 100 % and eliminating the page-table walk penalty. The author includes a C benchmark that allocates 1 GB with mmap using MAP_HUGETLB and shows the huge-page traversal completing in a fraction of the time required for 4 KB pages.

The Production Disaster

The author applied this theory to a typical microservice cluster: many small requests, high QPS, tiny per-request memory allocations, and fragmented access patterns. After disabling Transparent Huge Pages (THP) and reserving a large pool of static 2 MB huge pages via vm.nr_hugepages and vm.hugepagesz, TLB-miss counters did drop — but within ten minutes the system collapsed:

Memory usage jumped >30 % because every 1 KB allocation consumed a full 2 MB page, creating massive internal fragmentation. The reserved huge-page pool exhausted rapidly, and because static huge pages cannot be swapped or reclaimed, new allocations failed, triggering cascading timeouts.

CPU cache conflicts exploded. With 4 KB pages, different threads' data naturally spread across many pages, minimizing false sharing. Packing them into fewer 2 MB pages forced unrelated threads onto the same cache lines, amplifying MESI protocol traffic, lock contention, and soft-interrupt rates.

Page-fault latency increased. Although total fault count fell, each 2 MB fault (zeroing, disk I/O, merging) costs far more than a 4 KB fault, inflating P99/P999 latency.

Background khugepaged scanner continued running despite THP being disabled, scanning and merging pages under heavy lock contention, adding periodic CPU stalls.

The author reproduces the failure with a minimal C loop that repeatedly malloc(1024) / free — stable under 4 KB pages, instantly OOM under 2 MB pages — and a Spring @PostMapping snippet showing KB-sized per-request objects that each hijack a full huge page.

Rollback Procedure

Edit /etc/sysctl.conf: set vm.nr_hugepages = 0 and comment out vm.hugepagesz.

Run sudo sysctl -p.

Unmount the huge-page filesystem: sudo umount /dev/hugepages.

Reboot to ensure all settings take effect.

After rollback, average latency returned to ~550 ms and throughput recovered to 900+ records/sec.

Decision Framework: 4 KB vs. 2 MB

4 KB default pages — fine-grained allocation, near-zero internal fragmentation, low cache-line sharing, cheap page faults. Ideal for microservices, web apps, API gateways, and any workload with small, short-lived, fragmented memory access.

2 MB huge pages — TLB pressure relief for large, long-lived, contiguous memory regions. Suitable for databases (MySQL, PostgreSQL), distributed caches (Redis), big-data engines (Spark, Flink), HPC, DPDK packet processing, and media transcoding.

The closing principle: every performance knob is a trade-off; there is no universal optimization, only scenario-appropriate tuning.