Java OutOfMemoryError Deep Dive: Heap, Stack, Metaspace & Direct Memory

Java Heap Overflow

Java heap stores object instances; continuously creating objects until the heap reaches its maximum size triggers an OutOfMemoryError.

Simulation Code

<code>/**
 * VM Args：-Xms10m -Xmx10m -XX:+HeapDumpOnOutOfMemoryError
 */
public class HeapOOM {
    public static void main(String[] args) {
        List<byte[]> list = new ArrayList<>();
        while (true) {
            list.add(new byte[2048]);
        }
    }
}
</code>

Result:

<code>Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at cn.zhengsh.jvm.oom.HeapOOM.main(HeapOOM.java:16)
</code>

Problem Analysis

We need to determine whether the issue is a memory leak or a memory overflow.

Memory leak

Memory overflow

Memory Leak

Use the JDK tool

jvisualvm

to load heap dump files and analyze the reference chain from leaked objects to GC Roots. The following image illustrates the process.

Memory Overflow

If there is no leak, check the JVM heap parameters (-Xmx, -Xms) against the physical memory, and review code for long‑lived objects, excessive lifetimes, or inefficient data structures.

Virtual Machine Stack and Native Method Stack Overflow

HotSpot does not distinguish between the VM stack and the native method stack; the -Xoss option has no effect. Two exceptions are defined in the JVM specification:

If a thread requests a stack depth greater than the maximum allowed, a StackOverflowError is thrown.

If the VM allows dynamic stack expansion but cannot obtain enough memory, an OutOfMemoryError is thrown.

HotSpot does not support dynamic expansion, so only

StackOverflowError

occurs when the stack cannot accommodate a new frame.

StackOverflowError Example

<code>/** VM Args：-Xss128k */
public class JavaVMStackSOF {
    private int stackLength = 1;
    public void stackLeak() {
        stackLength++;
        stackLeak();
    }
    public static void main(String[] args) throws Throwable {
        JavaVMStackSOF oom = new JavaVMStackSOF();
        try {
            oom.stackLeak();
        } catch (Throwable e) {
            System.out.println("stack length:" + oom.stackLength);
            throw e;
        }
    }
}
</code>

Exception output:

<code>Exception in thread "main" java.lang.StackOverflowError
stack length:992
    at cn.zhengsh.jvm.oom.JavaVMStackSOF.stackLeak(JavaVMStackSOF.java:13)
    // ...
</code>

OutOfMemoryError Example

<code>public class JavaVMStackSOF2 {
    private static int stackLength = 0;
    public static void test() {
        long unused1, unused2, unused3, unused4, unused5, unused6, unused7, unused8, unused9, unused10,
             unused11, unused12, unused13, unused14, unused15, unused16, unused17, unused18, unused19, unused20,
             unused21, unused22, unused23, unused24, unused25, unused26, unused27, unused28, unused29, unused30,
             unused31, unused32, unused33, unused34, unused35, unused36, unused37, unused38, unused39, unused40,
             unused41, unused42, unused43, unused44, unused45, unused46, unused47, unused48, unused49, unused50,
             unused51, unused52, unused53, unused54, unused55, unused56, unused57, unused58, unused59, unused60,
             unused61, unused62, unused63, unused64, unused65, unused66, unused67, unused68, unused69, unused70,
             unused71, unused72, unused73, unused74, unused75, unused76, unused77, unused78, unused79, unused80,
             unused81, unused82, unused83, unused84, unused85, unused86, unused87, unused88, unused89, unused90,
             unused91, unused92, unused93, unused94, unused95, unused96, unused97, unused98, unused99, unused100;
        stackLength++;
        test();
        // assignments omitted for brevity
    }
    public static void main(String[] args) {
        try {
            test();
        } catch (Error e) {
            System.out.println("stack length:" + stackLength);
            throw e;
        }
    }
}
</code>

Result:

<code>stack length:6986
Exception in thread "main" java.lang.StackOverflowError
    at cn.zhengsh.jvm.oom.JavaVMStackSOF2.test(JavaVMStackSOF2.java:22)
    // ...
</code>

Summary

When a new stack frame cannot be allocated, HotSpot throws

StackOverflowError

. On JVMs that support dynamic stack expansion, the same code may lead to

OutOfMemoryError

.

Thread Creation Causing OOM

Running the following experiment may freeze the OS; execute it inside a virtual machine.

<code>/** VM Args：-Xss512k */
public class JavaVMStackOOM {
    private void dontStop() {
        while (true) { }
    }
    public void stackLeakByThread() {
        while (true) {
            Thread thread = new Thread(new Runnable() {
                @Override
                public void run() {
                    dontStop();
                }
            });
            thread.start();
        }
    }
    public static void main(String[] args) throws Throwable {
        JavaVMStackOOM oom = new JavaVMStackOOM();
        oom.stackLeakByThread();
    }
}
</code>

Method Area and Runtime Constant Pool Overflow

The method area stores class metadata. To overflow it, the program repeatedly generates classes using CGLib until Metaspace is exhausted.

<code>/** VM Args：-XX:MetaspaceSize=21m -XX:MaxMetaspaceSize=21m */
public class JavaMethodAreaOOM {
    public static void main(String[] args) {
        while (true) {
            Enhancer enhancer = new Enhancer();
            enhancer.setSuperclass(OOMObject.class);
            enhancer.setUseCache(false);
            enhancer.setCallback(new MethodInterceptor() {
                @Override
                public Object intercept(Object obj, Method method, Object[] args, MethodProxy proxy) throws Throwable {
                    return proxy.invokeSuper(obj, args);
                }
            });
            enhancer.create();
        }
    }
    static class OOMObject { }
}
</code>

Output:

<code>Caused by: java.lang.OutOfMemoryError: Metaspace
</code>

Constant Pool Example

The

String.intern()

method stores strings in the runtime constant pool. In JDK 6 the pool resides in the permanent generation; in JDK 7 it moved to the heap. The following code demonstrates the difference.

<code>public class RuntimeConstantPoolOOM2 {
    public static void main(String[] args) {
        String str1 = new StringBuilder("计算机").append("软件").toString();
        System.out.println(str1.intern() == str1);
        String str2 = new StringBuilder("ja").append("va").toString();
        System.out.println(str2.intern() == str2);
    }
}
</code>

On JDK 6 both prints are false; on JDK 7 the first is true and the second is false because “java” already exists in the pool.

Direct Memory Overflow

Direct memory size is controlled by

-XX:MaxDirectMemorySize

. The program obtains an

Unsafe

instance via reflection and repeatedly allocates 1 MB blocks until an

OutOfMemoryError

is thrown.

<code>/** VM Args：-Xmx20M -XX:MaxDirectMemorySize=10M */
public class DirectMemoryOOM {
    private static final int _1MB = 1024 * 1024;
    public static void main(String[] args) throws Exception {
        Field unsafeField = Unsafe.class.getDeclaredFields()[0];
        unsafeField.setAccessible(true);
        Unsafe unsafe = (Unsafe) unsafeField.get(null);
        while (true) {
            unsafe.allocateMemory(_1MB);
        }
    }
}
</code>

Exception output:

<code>Exception in thread "main" java.lang.OutOfMemoryError
    at java.base/jdk.internal.misc.Unsafe.allocateMemory(Unsafe.java:616)
    // ...
</code>

References

《深入理解 JVM 虚拟机-第三版》 周志明

https://docs.oracle.com/javase/specs/jls/se8/html/index.html