Why Did Upgrading to JDK 11 Spike Direct Memory Usage? A Deep Dive

Background

After upgrading the application to JDK 11, an alert was triggered by Sunfire indicating high memory utilization. The usage grew slowly after each restart and spiked after a period without restart. Initial suspicion fell on heap memory, but investigation revealed the impact of off‑heap (Direct) memory.

Environment

JDK version: ajdk11_11.0.14.13_fp2 Netty version:

4.1.31.Final

Problem Analysis

Using free -m showed a 76.5% memory utilization on the host, which differed from Sunfire’s 82% because Sunfire reports container‑level metrics. Monitoring the business container revealed that Java processes consumed about 74.3% of memory.

Java Heap Memory

Heap memory stores all Java objects created with new. It is managed by the JVM garbage collector and configured via -Xms (initial heap size) and -Xmx (maximum heap size). The heap is divided into Young Generation and Old Generation.

Non‑Heap / Off‑Heap Memory

Non‑heap memory includes the method area, Java stack, and native heap. Direct (off‑heap) memory is allocated via java.nio.ByteBuffer.allocateDirect() and is not subject to GC, reducing Full‑GC pauses but requiring manual lifecycle management. It can be queried through JMX buffers java.nio:type=BufferPool,name=direct and java.nio:type=BufferPool,name=mapped.

Reserved / Committed / Resident Memory

Reserved – virtual address space reserved by the OS but not yet backed by physical RAM. Committed – memory that the OS has actually allocated (physical RAM). Resident – portion of committed memory currently residing in RAM (not swapped).

The alert in this article monitors the Resident memory metric.

Why JDK 11 Upgrade Caused the Spike

In JDK 8 the flag USE_DIRECT_BUFFER_NO_CLEANER was true, using the allocateDirectNoCleaner path that avoids invoking a Cleaner. JDK 11 switches to the hasCleaner path, which calls UNSAFE.setMemory and forces the allocated memory to become Resident, dramatically increasing off‑heap usage.

if (maxDirectMemory == 0 || !hasUnsafe() || !PlatformDependent0.hasDirectBufferNoCleanerConstructor()) {
    USE_DIRECT_BUFFER_NO_CLEANER = false;
} else {
    USE_DIRECT_BUFFER_NO_CLEANER = true;
}

The change is caused by PlatformDependent0.hasDirectBufferNoCleanerConstructor() returning false on JDK 11 because reflective access to the DirectBuffer constructor is disabled unless the system property io.netty.tryReflectionSetAccessible is set to true.

How to Keep JDK 11 Using the NoCleaner Strategy

Add the following JVM arguments:

-Dio.netty.tryReflectionSetAccessible=true

--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED

--add-opens=java.base/java.nio=ALL-UNNAMED

After applying these parameters, Resident memory dropped by about 0.5 GB, matching the reduction in Direct memory usage.

Monitoring Before and After

Before: Resident memory ~4.8 GB; Direct memory ~526 MB.

After: Resident memory ~4.2 GB; Direct memory ~10 MB.

Arthas MBean queries confirm the Direct memory drop from ~551 MB to ~10 MB, while Mapped memory remains negligible.

Conclusion

JDK 11’s default Direct Memory allocation strategy can cause significant Resident memory growth. By enabling the NoCleaner path through JVM flags and opening necessary modules, the off‑heap memory footprint can be reduced, eliminating the memory‑utilization alerts observed after the upgrade.