How to Diagnose and Fix Online Service Failures: A Step‑by‑Step Checklist

CPU

Start by locating the problematic process with ps (or top to see high‑CPU processes). Use top -H -p <pid> to list threads, convert the thread ID to hexadecimal with

printf '%x
' <pid>

, and then run jstack <pid> | grep 'nid' -C5 --color to view the stack of the offending thread. Analyze the thread states; a large number of WAITING or TIMED_WAITING threads often indicates an issue. The jstack output can reveal dead loops, excessive context switches, or other logic errors.

Frequent GC

Check GC frequency with jstat -gc <pid> 1000, where the columns S0C/S1C, EC/EU, OC/OU, MC/MU show survivor, Eden, old‑gen, and metaspace usage. YGC/YGT, FGC/FGCT, GCT indicate young‑GC and full‑GC counts and times. If GC is unusually frequent, investigate further.

Context Switches

Use vmstat to view the cs column (context switches). For a specific process, pidstat -w <pid> shows voluntary ( nvcswch) and involuntary ( cswch) switches.

Disk

Check filesystem space with df -hl. For performance, run iostat -d -k -x and examine the %util column and rrqm/s / wrqm/s to identify the saturated disk. Use iotop to find which process is doing I/O, then translate the task ID to a PID with readlink -f /proc/*/task/<tid>/../... Inspect the process’s I/O counters via cat /proc/<pid>/io and list open files with lsof -p <pid>.

Memory

Run free to get an overview. Common issues are OOM, GC‑related OOM, and off‑heap memory.

Heap OOM

Typical messages:

Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread

Indicates thread‑stack allocation failure; reduce Xss or increase OS limits ( /etc/security/limits.conf).

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Heap reached -Xmx; look for leaks with jstack and jmap, then consider raising Xmx.

Caused by: java.lang.OutOfMemoryError: Metaspace

Metaspace hit MaxMetaspaceSize; adjust XX:MaxPermSize (pre‑Java 8) or increase the limit.

StackOverflow

Message:

Exception in thread "main" java.lang.StackOverflowError

Thread stack exceeds Xss; increase Xss cautiously.

Heap Dump Analysis

Export a heap dump with jmap -dump:format=b,file=filename <pid>, open it in MAT, use Leak Suspects or Top Consumers to locate leaks, and examine thread overview for thread‑related problems.

Off‑Heap Memory

Errors such as OutOfDirectMemoryError or OutOfMemoryError: Direct buffer memory often stem from Netty or NIO usage. Inspect the process’s memory map with pmap -x <pid> | sort -rn -k3 | head -30. Capture raw memory with

gdb --batch --pid <pid> -ex "dump memory filename.dump <addr> <addr+size>"

and view with hexdump -C filename | less. Enable Native Memory Tracking (NMT) via -XX:NativeMemoryTracking=summary or detail, then baseline with jcmd <pid> VM.native_memory baseline and later diff with jcmd <pid> VM.native_memory detail.diff. Use strace -f -e "brk,mmap,munmap" -p <pid> for low‑level allocation tracing.

GC Issues

Enable detailed GC logs with

-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps

. For G1GC, add -XX:+UseG1GC. Frequent young GC suggests a small Eden or survivor space; adjust -Xmn or -XX:SurvivorRatio. Long young‑GC pauses require examining G1 phases (Root Scanning, Object Copy, Ref Proc) in the logs. Full GC may be triggered by concurrent‑mark failures, promotion failures, large‑object allocation failures, or explicit System.gc(). Mitigations include increasing heap size, raising -XX:G1ReservePercent, adjusting -XX:InitiatingHeapOccupancyPercent, or increasing -XX:ConcGCThreads. Dump heap before/after full GC with -XX:HeapDumpPath=/xxx/dump.hprof and jinfo -flag +HeapDumpBeforeFullGC <pid>, jinfo -flag +HeapDumpAfterFullGC <pid>.

Network

Network problems are often the most complex. Distinguish connection timeout, read/write timeout, and pool‑related timeouts. Keep client timeout shorter than server timeout. Use netstat -s | egrep "listen|LISTEN" and ss -lnt to view queue statistics. TCP SYN‑queue overflow and accept‑queue overflow generate RST packets; monitor with netstat -s (look for “overflowed” and “sockets dropped”). Adjust backlog via acceptCount (Tomcat) or acceptQueueSize (Jetty), and OS parameters somaxconn and tcp_max_syn_backlog.

RST Packets

RST indicates abnormal connection reset. Capture traffic with tcpdump -i en0 tcp -w xxx.cap and analyze in Wireshark to spot RST flags.

TIME_WAIT and CLOSE_WAIT

Check counts with

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

or ss -ant | awk '{++S[$1]} END {for(a in S) print a, S[a]}'. Reduce excessive TIME_WAIT by enabling net.ipv4.tcp_tw_reuse=1 and net.ipv4.tcp_tw_recycle=1, or lowering tcp_max_tw_buckets. CLOSE_WAIT often stems from applications not closing sockets properly; use jstack to find threads stuck in countdownlatch.await or similar blocking calls.