Why Switching Linux Page Size from 4KB to 2MB Can Crash Your Performance

Conclusion: 2 MB hugepages are not a universal performance cure; replacing the default 4 KB pages without evaluating the workload often leads to severe regressions.

Why hugepages are thought to boost performance

Hugepages reduce the number of page‑table entries and dramatically increase TLB hit rates because the CPU can cache fewer translations. The translation from virtual to physical address is performed by the MMU and cached in the TLB, which is fast but very small.

Problems with the default 4 KB pages

For 1 GB of memory, 4 KB pages require over 260 000 page‑table entries.

The TLB cannot hold that many entries, causing frequent TLB misses; each miss forces a multi‑level page‑table walk that costs dozens of CPU cycles.

What changes when using 2 MB hugepages

Each page covers 512 × more memory, so 1 GB needs only 512 entries.

TLB can cache almost all entries, yielding near‑100 % hit rates and almost zero address‑translation overhead.

These facts are demonstrated with a simple benchmark that traverses 1 GB of memory using #include <stdio.h> and #include <sys/mman.h>. The 4 KB version iterates with a 4 KB stride, while the 2 MB version uses a 2 MB stride. The output shows the 2 MB traversal completing in a fraction of the time taken by the 4 KB traversal.

Why the micro‑service workload still got slower

In a typical micro‑service cluster the workload consists of many tiny memory allocations (≈1 KB) and high‑frequency requests. Switching to 2 MB pages caused several side effects:

Each 1 KB allocation forced the kernel to reserve an entire 2 MB page, inflating memory usage by >30 % and creating massive internal fragmentation.

Large pages increased cache‑line sharing under the MESI protocol, leading to a surge in cache‑coherency traffic and higher CPU soft‑interrupt counts.

Although the total number of page‑faults dropped, the cost of handling a single 2 MB fault is far higher than a 4 KB fault, inflating tail‑latency (P99/P999) dramatically.

The kernel's khugepaged thread continued to scan and attempt to merge pages, adding lock contention and periodic performance jitter.

Code snippets illustrate the problem. A Java micro‑service method allocating three small objects per request runs fine on 4 KB pages but, when the system is forced to use 2 MB pages, each request consumes a full 2 MB page, quickly exhausting the pre‑allocated hugepage pool and causing allocation failures.

Rollback procedure

To recover, the article advises editing /etc/sysctl.conf to reset the hugepage settings:

# Disable hugepages
vm.nr_hugepages = 0
# Comment out the explicit page size to revert to 4KB
# vm.hugepagesz = 2MB

Then run sudo sysctl -p, unmount /dev/hugepages with sudo umount /dev/hugepages, and reboot. After the reboot, latency returned to ~550 ms and throughput recovered to >900 req/s, confirming the rollback.

Final trade‑off analysis

4 KB pages excel in scenarios with fragmented, small‑allocation workloads (micro‑services, web APIs) because they keep memory usage tight, avoid internal fragmentation, and minimise cache‑coherency overhead. 2 MB hugepages shine for workloads that hold large, contiguous memory regions for extended periods (databases, Redis, DPDK, big‑data processing, HPC), where the reduced page‑table walk and high TLB hit rate outweigh the increased allocation granularity.

The key takeaway is that performance tuning must be matched to the workload; blindly applying a "big page" optimization can degrade service stability and overall performance.