How to Diagnose and Reproduce Different Java OOM Errors

Java OOM Types Overview

The interviewer's mention of OOM can refer to several distinct Java errors: heap overflow (java.lang.OutOfMemoryError: Java heap space), permgen/metaspace overflow (java.lang.OutOfMemoryError: Permgen space), and native thread creation failure (java.lang.OutOfMemoryError: Unable to create new native thread).

1. Heap Overflow

Java heap stores object instances. When the heap size limit is reached, an OOM occurs. The article limits the heap to 20 MB (non‑expandable) and enables heap dump on OOM: -XX:+HeapDumpOnOutOfMemoryError Two sample cases demonstrate heap OOM. The first shows the typical error message "java.lang.OutOfMemoryError: Java heap space". To diagnose, use a heap‑dump analysis tool (e.g., jprofile) to determine whether the problem is a memory leak or pure overflow, then:

Check if leaked objects are truly needed and trace their reference chains to GC roots.

If no leak, verify JVM heap parameters (-Xmx, -Xms) against physical memory and review code for long‑lived objects or inefficient data structures.

2. Stack / Native Method Stack Overflow

HotSpot does not distinguish between the Java stack and the native method stack; the -Xoss flag exists but has no effect. Stack size is controlled only by -Xss. Two experiments are presented:

Reduce -Xss (e.g., to 160 k) to force a StackOverflowError . The stack depth reported in the error shrinks accordingly.

Define a method with a large number of local variables, increasing the frame size, which also triggers StackOverflowError on HotSpot.

When many threads are created, each thread’s stack consumes memory; if the total stack allocation exceeds available RAM, the JVM throws java.lang.OutOfMemoryError: unable to create native thread. The article shows that a thread that OOMs releases its heap memory, allowing other threads to continue running, but frequent OOMs cause heavy GC overhead.

3. Method Area / Runtime Constant Pool Overflow

The runtime constant pool resides in the method area. In JDK 6 it lives in the permanent generation (PermGen); from JDK 7 onward it moved to the heap (Metaspace). The article demonstrates OOM by repeatedly calling String.intern() to fill the constant pool:

Exception in thread "main" java.lang.OutOfMemoryError: PermGen space
    at java.lang.String.intern(Native Method)
    at org.fenixsoft.oom.RuntimeConstantPoolOOM.main(RuntimeConstantPoolOOM.java:18)

On JDK 7+ the same code does not overflow because the constant pool is on the heap. To limit the method area on newer JVMs, use Metaspace flags such as:

-XX:MetaspaceSize=10M
-XX:MaxMetaspaceSize=10M

Additional scenarios that can exhaust the method area include generating many dynamic classes (e.g., via CGLib, Groovy, OSGi, or JSP compilation). The article lists relevant JVM options for Metaspace tuning ( -XX:MetaspaceSize, -XX:MaxMetaspaceSize, -XX:MinMetaspaceFreeRatio, -XX:MaxMetaspaceFreeRatio).

4. Direct (Native) Memory Overflow

Direct memory size can be set with -XX:MaxDirectMemorySize; if omitted it defaults to the maximum heap size ( -Xmx). The article shows allocation via Unsafe.allocateMemory(), which throws OOM without producing a heap dump. Detecting this situation involves noticing a small heap‑dump file despite an OOM and confirming the use of NIO or other direct‑memory APIs.

Overall, the article provides a systematic approach to reproduce each OOM type, the JVM flags that influence them, and practical steps for diagnosis and mitigation.