How to Systematically Diagnose High RSS Memory Usage in Java Services

Introduction

Online problem troubleshooting is a low‑frequency but high‑urgency task, especially when dealing with memory‑related issues such as high RSS (Resident Set Size). Efficient diagnosis requires a clear thinking framework combined with appropriate tooling.

Step‑by‑step troubleshooting workflow

1. Verify whether heap memory is oversized

Check the Java process heap usage; a common rule of thumb is to keep heap memory below 75% of the physical RAM. Use jcmd <pid> GC.heap_info or visual monitoring tools. If the heap approaches the 75% threshold, reduce -Xmx or analyze the heap dump with MAT or JFR.

2. Detect excessive ARENA allocations

If heap size is normal, inspect the ARENA region, which can be examined without restarting the JVM. Run the following command:

sudo -u <user> pmap -x <pid> | sort -gr -k2 | less

Large blocks around 60 000 KB indicate ARENA pressure. Mitigate it by setting the environment variable: export MALLOC_ARENA_MAX=1 Only the value 1 is effective; larger numbers do not limit ARENA count.

3. Enable Native Memory Tracking (NMT) to expose non‑heap consumption

Add the JVM flag and restart the process: -XX:NativeMemoryTracking=detail After restart, query NMT data with:

sudo -u <user> jcmd <pid> VM.native_memory detail

The output shows memory usage across regions such as Class, Thread, Code, GC, Compiler, Internal, and Symbol. Pay special attention to Class, Thread, and GC, which can together consume a significant portion of memory.

4. Examine off‑heap memory usage

Off‑heap memory is often large in network‑intensive Java services that use NIO. Query DirectByteBuffer and MappedByteBuffer usage via JMX:

java.nio:name=direct,type=BufferPool

java.nio:name=mapped,type=BufferPool

If off‑heap consumption is excessive, limit it with the JVM option: -XX:MaxDirectMemorySize=<size> When off‑heap usage remains unexplained, consider using jemalloc for deeper analysis.

Tool‑centric automation

To reduce manual effort, the article proposes a shell‑based automation script that orchestrates the above steps. The script performs:

Discovery of Java process PID via jps -v Collection of physical memory stats ( free -m)

Heap inspection ( jcmd GC.heap_info)

GC metrics ( jstat -gcutil)

ARENA detection using the pmap pipeline

Native Memory Tracking activation and reporting

Off‑heap queries via JMX or jmxterm The script can be extended to a client‑server‑browser model, allowing engineers to run diagnostics without direct SSH access, which is useful in tightly controlled production environments.

Conclusion

Following the four‑step methodology—heap size check, ARENA analysis, native memory tracking, and off‑heap inspection—covers the majority of high‑RSS scenarios. Packaging these steps into reusable scripts creates a lightweight troubleshooting toolkit that can be expanded to cover CPU, disk, network, and GC issues using open‑source tools such as Arthas and Async‑Profiler.