How to Diagnose and Resolve Common Java Server Performance Issues

Overview

Online incidents often involve CPU, disk, memory, and network problems; most issues span multiple layers, so a systematic four‑step investigation (CPU → Disk → Memory → Network) is recommended.

CPU

Start by checking CPU usage. CPU anomalies are usually easier to locate. Common causes include business‑logic loops, frequent GC, and excessive context switches; the most frequent cause is problematic business or framework logic, which can be examined with jstack.

Using jstack to analyze CPU problems

Find the process PID with ps (or top to see which process consumes the most CPU). Then run: top -H -p pid to identify high‑CPU threads. Convert the thread ID to hexadecimal:

printf '%x
' pid

Resulting nid is used to search the jstack output: jstack pid | grep 'nid' -C5 --color Focus on threads in WAITING or TIMED_WAITING states; BLOCKED threads are less common. For a quick overview of thread states, run:

cat jstack.log | grep "java.lang.Thread.State" | sort -nr | uniq -c

Frequent GC

Use jstat -gc pid 1000 to monitor GC generation changes (sampling interval 1000 ms). Columns S0C/S1C, S0U/S1U, EC/EU, OC/OU, MC/MU represent Survivor, Eden, Old, and Metaspace capacities and usage. YGC/YGT, FGC/FGCT, GCT show Young GC, Full GC counts and times. If GC appears too frequent, investigate further with dump analysis.

Context Switches

Inspect context switches with vmstat. The cs column shows the number of switches. To monitor a specific PID, use: pidstat -w pid Columns cswch and nvcswch indicate voluntary and involuntary switches.

Disk

Disk issues are also fundamental. Check disk space with:

df -hl

Performance problems can be diagnosed with:

iostat -d -k -x

The %util column shows disk write intensity; rrqm/s and wrqm/s indicate read/write speeds, helping locate the problematic disk. Identify the responsible process with iotop or by converting a thread ID to PID via readlink -f /proc/*/task/tid/../.., then inspect I/O with:

cat /proc/pid/io

List open files with lsof -p pid.

Memory

Memory issues are more complex and include OOM, GC problems, and off‑heap memory. Start with free to view overall memory status.

Heap Memory OOM

Typical OOM messages:

Exception in thread “main” java.lang.OutOfMemoryError: unable to create new native thread – insufficient native memory for thread stacks; check thread pools, use jstack / jmap, or increase OS limits.

Exception in thread “main” java.lang.OutOfMemoryError: Java heap space – heap reached -Xmx limit; look for leaks with jstack / jmap, then consider increasing -Xmx.

Exception in thread “main” java.lang.OutOfMemoryError: Metaspace – metaspace reached -XX:MaxMetaspaceSize; adjust with -XX:MaxPermSize for older JVMs.

Stack Overflow

Indicates thread stack exceeds -Xss. Reduce -Xss or investigate code paths.

Using JMAP to locate memory leaks

Export a heap dump:

jmap -dump:format=b,file=filename pid

Analyze the dump with MAT (Memory Analyzer Tool), focusing on “Leak Suspects” or “Top Consumers”.

GC Issues

GC problems can cause CPU load and memory pressure. Enable detailed GC logging with

-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps

. Analyze Young GC frequency with jstat; if too frequent, consider increasing -Xmn or -XX:SurvivorRatio. For long GC pauses, examine G1 log phases such as Root Scanning, Object Copy, and Ref Proc.

Full GC triggers include concurrent marking failure, promotion failure, large object allocation failure, or explicit System.gc(). Dump heap before/after Full GC with -XX:HeapDumpPath and inspect with jinfo or jmap.

jinfo -flag +HeapDumpBeforeFullGC pid
jinfo -flag +HeapDumpAfterFullGC pid

Network

Network problems are complex and often the hardest to diagnose.

Timeouts

Distinguish between connection timeout and read/write timeout. Keep client timeout smaller than server timeout to avoid hanging connections.

TCP Queue Overflow

Two queues exist: SYN (half‑open) and accept (full‑open). If the accept queue is full during the third handshake, the server may drop the ACK or send an RST depending on tcp_abort_on_overflow. Monitor overflow with:

netstat -s | egrep "listen|LISTEN"

Check queue sizes with ss -lnt and adjust OS parameters net.ipv4.tcp_tw_reuse, net.ipv4.tcp_tw_recycle, or tcp_max_tw_buckets as needed.

# enable reuse of TIME‑WAIT sockets
net.ipv4.tcp_tw_reuse = 1
# enable fast recycle of TIME‑WAIT sockets
net.ipv4.tcp_tw_recycle = 1

RST Packets

RST indicates an abnormal connection reset, often caused by sending data to a closed socket or by queue overflows. Use tcpdump and Wireshark to capture and analyze RST packets.

tcpdump -i en0 tcp -w capture.cap

TIME_WAIT and CLOSE_WAIT

TIME_WAIT ensures delayed packets are handled and prevents premature RSTs; excessive TIME_WAIT can be mitigated by enabling reuse/recycle as above. CLOSE_WAIT usually results from applications not closing sockets properly; investigate with jstack to find threads stuck in I/O or waiting on latches.