Java OOM Troubleshooting: Heap, Metaspace, Stack & Direct Memory Leaks - Complete Guide

Core Insight: OOM Is Not a Single Failure but Four Distinct Categories

Most developers mistakenly treat OOM as simply "out of memory," but production OOM errors fall into four types, each with unique root causes, investigation methods, and fixes. Blindly increasing heap size only masks leaks and delays crashes.

java.lang.OutOfMemoryError: Java heap space — Heap overflow (90% of production OOMs)

java.lang.OutOfMemoryError: Metaspace — Metaspace overflow (dynamic class loading, hot reload, class leaks)

java.lang.StackOverflowError — Stack overflow (infinite recursion, excessive call depth)

java.lang.OutOfMemoryError: Direct buffer memory — Direct memory overflow (Netty, NIO buffers not released)

Reading the error keyword immediately narrows the investigation scope.

Most Frequent: Heap Overflow (Java heap space)

Heap OOM splits into two fundamentally different scenarios requiring different fixes:

Scenario 1: Instantaneous Large-Object Spike

Symptom: Service runs normally, then suddenly crashes with OOM; restart yields long-term stability.

Root cause: A single request loads massive data — unpaginated bulk queries, full-table scans, huge file/JSON parsing — instantly filling the heap.

Scenario 2: Gradual Memory Leak (Most Difficult)

Symptom: Service starts fine, slows over 1–3 days, old-gen memory only grows, eventually OOM; restart temporarily restores.

Root cause: Objects remain permanently reachable and cannot be GC'd — a code-level leak bug. Increasing heap only postpones the crash.

Enterprise-Grade Heap OOM Investigation SOP

Core flow: Capture heap dump → Analyze large objects → Identify resident objects → Trace back to code → Fix

Step 1: Pre-configure JVM to Auto-Save OOM Snapshot

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/heapdump.hprof

This preserves the crash scene automatically.

Step 2: Manual Heap Dump During Rising Memory (Pre-Crash)

jmap -dump:format=b,file=heap.hprof <PID>

Run during low traffic to avoid STW pauses.

Step 3: Analyze .hprof with MAT (Memory Analyzer Tool)

Focus on three key views:

Dominator Tree — Find the largest retained objects and their origins.

Leak Suspects — MAT's automated leak detection.

Histogram — Count instances and memory share per class; spot many duplicate resident objects.

Five Root Causes Covering 99% of Java Heap Leaks

ThreadLocal not removed — Thread reuse in pools keeps objects alive indefinitely.

Static collections growing without bound — static List/Map with no cleanup or expiration.

Unclosed connection resources — DB connections, IO streams, Redis/HTTP clients left open.

Caches lacking expiration or eviction — Local caches accumulate forever.

Scheduled tasks repeatedly creating objects — High-frequency jobs allocate new objects each run without reuse.

Metaspace Overflow (OutOfMemoryError: Metaspace)

Metaspace holds class metadata, method data, constant pool, and dynamically generated classes. It defaults to unlimited (bounded by physical RAM); overflow crashes the JVM.

Key Symptoms

Metaspace usage climbs continuously; Full GC frequency spikes.

Error: OutOfMemoryError: Metaspace; service dies.

Heap usage normal; only metaspace exhausted.

Core Causes

Dynamic class generation: proxies, CGLIB, bytecode frameworks.

Hot deploy/reload loading new classes without unloading old ones.

Massive JSP compilation or temporary class generation.

MaxMetaspaceSize set too low for the class-load volume.

Solutions

-XX:MaxMetaspaceSize=512m

Root fix: Reduce dynamic class generation, disable unused hot-reload, fix code that creates excessive proxy classes.

Stack Overflow (StackOverflowError)

Pure code-level bug; easiest to diagnose because the stack trace prints the exact recursive chain.

Causes

Recursion without termination condition or infinite recursive loop.

Method nesting exceeds JVM max frame depth.

Loop with unbounded nested calls.

Fix

Inspect the stack trace, locate the recursive method, add proper termination condition and boundary checks. No JVM tuning needed.

Direct Memory Overflow (OutOfMemoryError: Direct buffer memory)

Common in Netty, Dubbo, NIO, file-transfer workloads.

Cause

Off-heap ByteBuffer / DirectBuffer allocated via NIO/Netty not released after use; GC cannot reclaim them, leading to accumulation.

Solutions

Explicitly release every ByteBuffer / DirectBuffer after use.

Enable Netty leak detection; optimize buffer reuse logic.

Cap direct memory:

-XX:MaxDirectMemorySize=256m

Production OOM Emergency + Root-Cause SOP (6 Steps)

Read the error log — Distinguish heap, metaspace, stack, or direct memory OOM.

Preserve the scene — Grab .hprof, GC logs, thread dumps; do not restart immediately .

Stop the bleeding — Restart, degrade traffic, scale out to restore availability.

Deep MAT analysis — Pinpoint large/resident/leaking objects; trace to business code.

Code-level cure — Fix leak bugs, add resource cleanup, implement expiration, optimize large queries.

Monitoring safety net — Alert on heap/old-gen/metaspace trends; catch leaks early.

Long-Term Prevention Rules (Enforce in Code Reviews)

All queries must paginate — Ban full-table scans and bulk loads.

Release immediately after use — ThreadLocal, streams, connections, buffers.

Local caches must expire — Every cache/static collection needs TTL, eviction, or clear policy.

Recursion must have termination — Enforce boundary checks on all recursive logic.

Production must enable heap dump on OOM — Uniform JVM flag across all services.

Watch GC trends daily — Rising old-gen curve = leak warning; optimize proactively.

Summary

This guide covers all four OOM categories, shattering the "OOM = heap full" misconception. It delivers a closed-loop capability: from log classification, snapshot capture, root-cause analysis, emergency response, to code-level cure. Independent OOM troubleshooting is a core skill separating junior from mid-level engineers and is essential for production stability.