Why System.gc Matters for Java NIO Memory Management

Based on OpenJDK 17, this article discusses why System.gc frequently appears in JDK NIO off‑heap memory allocation scenarios such as FileChannel#map and ByteBuffer.allocateDirect.

When a memory‑mapped file operation runs out of virtual address space, the JVM throws OutOfMemoryError, catches it, invokes System.gc to try to free native memory, and retries the mapping; if it still fails, an IOException is thrown.

private Unmapper mapInternal(MapMode mode, long position, long size, int prot, boolean isSync) throws IOException {
    try {
        // If map0 did not throw an exception, the address is valid
        addr = map0(prot, mapPosition, mapSize, isSync);
    } catch (OutOfMemoryError x) {
        // An OutOfMemoryError may indicate that we have exhausted memory,
        // so force gc and re‑attempt map
        System.gc();
        try {
            Thread.sleep(100);
        } catch (InterruptedException y) {
            Thread.currentThread().interrupt();
        }
        try {
            addr = map0(prot, mapPosition, mapSize, isSync);
        } catch (OutOfMemoryError y) {
            // After a second OOME, fail
            throw new IOException("Map failed", y);
        }
    }
}

Similarly, ByteBuffer.allocateDirect first checks the current off‑heap usage via Bits.reserveMemory against the -XX:MaxDirectMemorySize limit; if the limit is exceeded, the JVM attempts to release native memory held by already reclaimed direct buffers before calling System.gc.

class Bits {
    static void reserveMemory(long size, long cap) {
        // ... omitted ...
        // If the requested off‑heap capacity exceeds -XX:MaxDirectMemorySize,
        // return false to indicate the request cannot be satisfied
        if (tryReserveMemory(size, cap)) {
            return;
        }
        // Try to free native memory behind reclaimed direct buffers
        System.gc();
        Thread.sleep(sleepTime);
        // ... omitted ...
    }
}

Calling System.gc deliberately in NIO is necessary because DirectByteBuffer and its subclass MappedByteBuffer allocate native memory outside the Java heap, which the regular GC does not manage. When a DirectByteBuffer becomes unreachable, its Cleaner is enqueued; after a full GC the ReferenceHandler thread runs the Cleaner to release the native memory.

In practice, DirectByteBuffer objects are small on the Java heap but may hold large native memory regions, so they often survive to the old generation and are not reclaimed promptly, leading to native memory leaks and possible kernel OOM.

The HotSpot implementation of System.gc ultimately calls Runtime.gc(), a native method that invokes JVM_GC(). If the flag -XX:+DisableExplicitGC is set, the call is ignored; otherwise it triggers a full GC with cause _java_lang_system_gc.

JVM_ENTRY_NO_ENV(void, JVM_GC(void)) {
    if (!DisableExplicitGC) {
        EventSystemGC event;
        event.set_invokedConcurrent(ExplicitGCInvokesConcurrent);
        Universe::heap()->collect(GCCause::_java_lang_system_gc);
        event.commit();
    }
}

The concrete heap type depends on the selected garbage collector. For SerialGC, ParallelGC, ZGC, G1, and Shenandoah, System.gc ultimately calls the collect method of the corresponding heap class, which may perform a stop‑the‑world full GC or a concurrent full GC depending on JVM flags such as -XX:+ExplicitGCInvokesConcurrent.

If -XX:+DisableExplicitGC is enabled, System.gc does nothing.

With SerialGC, ParallelGC, or ZGC, System.gc triggers an immediate stop‑the‑world full GC.

With G1, CMS, or Shenandoah and -XX:+ExplicitGCInvokesConcurrent, System.gc initiates a concurrent full GC; otherwise it falls back to a stop‑the‑world full GC.

Understanding this behavior helps developers decide when it is safe or necessary to invoke System.gc in performance‑critical NIO code.