Systematic Approach to Online RSS Memory Usage Troubleshooting and Tool Automation

Online problem troubleshooting, especially for high RSS (Resident Set Size) usage, is a low‑frequency but critical task that requires an efficient, systematic approach.

The article first explains the importance of a clear diagnostic mindset and lists popular open‑source and commercial tools such as Arthas, Async‑Profiler, JMC, MAT, and commercial profilers like jProfiler and YourKit.

It then introduces a four‑step troubleshooting framework using RSS high memory as an example:

2.1 Heap memory too large – Verify heap usage via jcmd <pid> GC.heap_info and ensure it does not exceed ~75% of physical memory; adjust -Xmx or analyze heap dumps with MAT/JFR if necessary.

2.2 Excessive ARENA regions – Run

sudo -u <user> pmap -x <pid> | sort -gr -k2 | less

to spot many 60‑KB regions and limit them by setting export MALLOC_ARENA_MAX=1.

2.3 Native memory tracking (NMT) overload – Enable NMT with -XX:NativeMemoryTracking=detail, restart the JVM, and query details via jcmd <pid> VM.native_memory detail to identify large JVM internal allocations such as Class, Thread, GC, and Code.

2.4 Off‑heap memory excess – Check DirectByteBuffer and MappedByteBuffer usage via JMX or jmxterm; control it with -XX:MaxDirectMemorySize or employ jemalloc for deeper analysis.

The article then discusses automating these steps into a shell‑based toolchain, outlining the workflow for each sub‑step, required commands (e.g., jps -v, free -m, jstat -gcutil), and how to package the process into a script that can be executed with minimal manual intervention.

Finally, it summarizes that the described methodology and tooling significantly improve troubleshooting efficiency, can be extended to other resource issues (CPU, disk, network, GC), and suggests evolving the solution into a client‑server‑browser system for environments with strict access controls.