Master Java Performance Debugging: Hprof, pidstat, and Real‑World Memory Leak Fixes

Developer Self‑Testing with Hprof

A simple Java program that sleeps is used to illustrate performance testing. Run the program with the JVM argument

-agentlib:hprof=cpu=samples,depth=10,format=a,file=java.hprof.txt

. The agent creates java.hprof.txt, a text report that lists each class method together with the amount of CPU time spent in that method, allowing quick identification of hot spots.

Hprof Options

Hprof is a Java agent, not a standalone profiler. Use java -agentlib:hprof=help to display the full option list. Common options include:

-agentlib:hprof=cpu=samples,depth=10,format=a,file=java.hprof.txt

– sample CPU usage and write a text report. -agentlib:hprof=heap,format=b,file=/test.hprof – generate a binary heap dump. -agentlib:hprof=heap,format=a,file=heap.txt – generate a text heap dump.

Because Hprof runs as a Java agent, it can be combined with JUnit tests to catch performance regressions early.

Per‑Thread CPU Monitoring with pidstat

After executing the same test program, the command pidstat -p <pid> 1 reports CPU usage for each thread every second. The %usr column shows user‑mode CPU consumption. In the example, thread ID 855 consumes a large share of CPU. Converting 855 to hexadecimal (0x357) and searching for that value in the application log ( testlog.txt) pinpoints the exact code path responsible for the bottleneck.

Obtaining pidstat

pidstat is part of the Sysstat suite. Source code can be downloaded from http://sebastien.godard.pagesperso-orange.fr/download.html and built with the usual ./configure && make && make install sequence. Once installed, pidstat can monitor both process‑level and thread‑level performance.

Key pidstat Fields

PID – process identifier.

%usr – CPU usage in user mode.

%system – CPU usage in kernel mode.

%guest – CPU time spent in virtual machines.

%CPU – total CPU usage (adjusted for SMP with -I).

CPU – processor number on which the task is running.

Command – command name.

Common I/O fields:

kB_rd/s – kilobytes read per second.

kB_wr/s – kilobytes written per second.

kB_ccwr/s – kilobytes of cancelled writes per second.

Memory‑Leak Case Study

In an eight‑node cluster, one node intermittently throws java.lang.OutOfMemoryError: GC overhead limit exceeded followed by java.lang.OutOfMemoryError: Java heap space. The GC overhead limit is triggered when the garbage collector spends more than 98 % of wall‑clock time reclaiming less than 2 % of the heap, a safeguard introduced in HotSpot 1.6.

A heap dump captured at the failure point is opened in VisualVM. The class histogram shows that char[], LinkedHashMap, and String objects occupy the largest portion of memory, although their instance counts are not unusually high. Further inspection reveals that a middleware component caches a large number of URL strings generated by Hessian and never releases them, causing sustained memory pressure.

Remediation Steps

Add the JVM flag -XX:-UseGCOverheadLimit to disable the pre‑emptive alarm, allowing the application to fail with a full OOM instead of a premature warning.

Audit the codebase for large, long‑living allocations, unreleased resources, or loops that retain objects, and introduce proper cache eviction or object release logic.