Understanding Java Heap, Stack, Metaspace, and Direct Memory OutOfMemoryError with Examples

Java Heap Overflow

The Java heap stores object instances; continuously creating objects until the heap reaches its maximum capacity triggers an OutOfMemoryError. Common causes are large containers like Map or List that cannot be reclaimed.

Memory overflow: insufficient memory space prevents allocation of new objects.

Memory leak: objects that should be released remain referenced, often due to containers holding them.

When the following message appears, a heap overflow has occurred:

java.lang.OutOfMemoryError: Java heap space

Ensure that GC roots have reachable paths to objects so the garbage collector can reclaim them.

Example

Set JVM memory parameters:

-verbose:gc -Xms20m -Xmx20m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\dump

Sample code that continuously adds objects to a list until the heap is exhausted:

/**
 * java heap memory overflow
 * VM Args: -Xms20m -Xmx20m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\dump
 */
public class HeapOutOfMemoryErrorTest {
    public static void main(String[] args) throws InterruptedException {
        List
list = Lists.newArrayList();
        for (long i = 1; ; i++) {
            list.add(new Object());
            if (i % 100_000 == 0) {
                System.out.println(Thread.currentThread().getName() + "::" + i);
            }
        }
    }
}

Running the program produces GC logs and eventually the exception:

[GC (Allocation Failure)  5596K->1589K(19968K), 0.0422027 secs]
main::100000
...
java.lang.OutOfMemoryError: Java heap space
Dumping heap to D:\dump\java_pid7432.hprof ...

Analysis Tools

The JDK tool jvisualvm.exe can analyze .hprof and .dump files. Identify the largest objects, verify their necessity, and adjust JVM memory size accordingly. Use GC Roots to trace reference chains and locate leaking code.

1. Find the largest object

2. Locate the specific object

Solutions for Heap Overflow

Optimize code to remove large objects.

Adjust JVM memory size (‑Xmx and ‑Xms).

JVM Stack and Native Method Stack Overflow

If a thread requests a stack depth greater than the VM allows, a StackOverflowError is thrown.

If the VM cannot allocate enough memory while expanding the stack, an OutOfMemoryError occurs.

These two descriptions overlap because both indicate that the stack cannot be extended further.

StackOverflowError

Main causes:

Single thread requests a stack depth exceeding the VM limit.

Too many threads are created.

Excessive Stack Depth in a Single Thread

Typical reasons include recursion, circular dependencies, or very deep call chains (e.g., decorator patterns).

Exception message:

java.lang.StackOverflowError

Decorator example:

Collections.unmodifiableList(
    Collections.unmodifiableList(
        Collections.unmodifiableList(
            Collections.unmodifiableList(
                Collections.unmodifiableList(
                    ...)))));

Recursive example:

/**
 * Java virtual machine stack and native method stack overflow test
 * VM Args: -Xss128k
 */
public class StackOverflowErrorErrorTest {
    private int stackLength = 0;
    public void stackLeak() {
        stackLength++;
        stackLeak();
    }
    public static void main(String[] args) {
        StackOverflowErrorErrorTest sof = new StackOverflowErrorErrorTest();
        try {
            sof.stackLeak();
        } catch (Exception e) {
            System.out.println(sof.stackLength);
            e.printStackTrace();
        }
    }
}

Running with default stack size yields a depth of about 1,300; increasing -Xss1m raises the depth to over 20,000.

Too Many Threads

Creating many threads can exhaust native memory, leading to:

java.lang.OutOfMemoryError: unable to create new native thread

/**
 * Java stack and native method stack overflow test by creating many threads
 * VM Args: -verbose:gc -Xss20M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\dump
 */
public class StackOutOfMemoryErrorTest {
    private static int threadCount;
    public static void main(String[] args) throws Throwable {
        try {
            while (true) {
                threadCount++;
                new Thread(() -> {
                    try { Thread.sleep(1000L * 60 * 10); } catch (InterruptedException e) { e.printStackTrace(); }
                }).start();
            }
        } catch (Throwable e) {
            e.printStackTrace();
            throw e;
        } finally {
            System.out.println("threadCount=" + threadCount);
        }
    }
}

On a 64‑bit machine the test may not reproduce, but on a 32‑bit system it can quickly exhaust memory.

Metaspace (Method Area) Overflow

The method area stores class metadata. Generating a large number of classes (e.g., via CGLIB proxies, many JSPs, or OSGi) can fill this space.

/**
 * Java method area / metaspace memory overflow
 * VM Args JDK 1.8: -verbose:gc -Xmx20m -XX:MetaspaceSize=5m -XX:MaxMetaspaceSize=5m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\dump
 */
public class MethodAreaOutOfMemoryErrorTest {
    static class MethodAreaOOM {}
    public static void main(String[] args) {
        while (true) {
            Enhancer enhancer = new Enhancer();
            enhancer.setSuperclass(MethodAreaOOM.class);
            enhancer.setCallback(new MethodInterceptor() {
                @Override
                public Object intercept(Object obj, Method method, Object[] params, MethodProxy proxy) throws Throwable {
                    return proxy.invokeSuper(obj, params);
                }
            });
            enhancer.create();
        }
    }
}

Resulting exception:

java.lang.OutOfMemoryError: Metaspace
...

Runtime Constant Pool Overflow

Calling String.intern() repeatedly can fill the runtime constant pool, which in JDK 1.6 resides in the method area.

/**
 * Java runtime constant pool overflow
 * VM Args JDK 1.6: -verbose:gc -XX:PermSize=10m -XX:MaxPermSize=10m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\dump
 */
public class RuntimeConstantOutOfMemoryErrorTest {
    public static void main(String[] args) {
        List
list = new ArrayList<>();
        for (int i = 0; ; i++) {
            list.add(String.valueOf(i).intern());
        }
    }
}

Resulting exception:

java.lang.OutOfMemoryError: PermGen space
    at java.lang.String.intern(Native Method)
    ...

Direct Memory Overflow

Direct memory size can be set with -XX:MaxDirectMemorySize ; if unspecified it defaults to the maximum heap size.

/**
 * Java direct memory overflow
 * VM Args JDK 1.6: -verbose:gc -Xms20m -XX:MaxDirectMemorySize=10m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\dump
 */
public class DirectMemoryOutOfMemoryErrorTest {
    public static void main(String[] args) throws IllegalAccessException {
        int _1M = 1024 * 1024;
        Field unsafeField = Unsafe.class.getDeclaredFields()[0];
        unsafeField.setAccessible(true);
        Unsafe unsafe = (Unsafe) unsafeField.get(null);
        while (true) {
            unsafe.allocateMemory(_1M);
        }
    }
}

Resulting exception:

java.lang.OutOfMemoryError
    at sun.misc.Unsafe.allocateMemory(Native Method)
    ...

When a DirectMemory OOM occurs, the heap dump file is usually small because the overflow is outside the heap.

General Solutions

Use -XX:MaxDirectMemorySize to limit direct memory.

Adjust JVM memory options ( -Xmx , -Xms , -XX:MetaspaceSize , etc.).

Optimize code to avoid large containers, deep recursion, or excessive thread creation.

Disable GC overhead limit check with -XX:-UseGCOverheadLimit only as a temporary measure.

Source code repository: https://github.com/wyh-spring-ecosystem-student/spring-boot-student/tree/releases