Skip to content
JVM Diagnostic Toolchain: Snapshots, Profilers & Arthas — Costs, Mechanisms & Production SOPs

JVM Diagnostic Toolchain: Snapshots, Profilers & Arthas — Costs, Mechanisms & Production SOPs

This article analyzes the JVM diagnostic toolchain across three layers — snapshot tools (jstack, jmap, jcmd), interactive arthas, and continuous profilers (async-profiler, JFR) — explaining each tool's observation mechanism, performance cost, and correct usage order through real incidents, source code, and production SOPs for CPU, memory, and thread issues.

Architecture Pony
Architecture Pony
Architecture Pony

Three-Layer Diagnostic Model and Tool Costs

The article opens with three production incidents that illustrate why tool choice and order matter:

Incident 1: CPU at 95% during a big promotion; repeated jstack snapshots showed only normal stacks. A 60-second async-profiler run revealed a broad plateau in the flame graph: logback synchronous appender lock contention, chaining dozens of business threads. The lesson: a single snapshot (a "shutter") misses millisecond-level hotspots; continuous sampling (a "video") catches them.

Incident 2: Metaspace slowly rising. The engineer ran jmap -dump:live on an 8 GB heap, triggering a Full GC and a 30-second STW pause — worse than the leak itself. The correct first step is jcmd GC.class_histogram, a seconds-level lightweight histogram that identifies the largest classes without a full dump.

Incident 3: Intermittent 3-second hangs. A single jstack showed all threads WAITING. Five snapshots at 200 ms intervals revealed 8 threads repeatedly BLOCKED on the same monitor; arthas trace then exposed a slow SQL query.

The author organizes tools into three layers by intervention depth:

L1 Snapshot Layer (jstack, jmap, jstat, jcmd): answers "what is the state right now?" All require a safepoint — the JVM pauses all threads at a consistent state. Cost = time to reach safepoint + capture time. jstack is milliseconds; histogram is seconds; heap dump is tens of seconds proportional to heap size.

L2 Interactive Layer (arthas): answers "what happens during execution?" Uses the same attach mechanism (safepoint) but stays resident, supports multi-shot aggregation. Each enhancement is bytecode surgery — suitable for low-frequency, targeted, short-term deep dives, not continuous monitoring.

L3 Profiling Layer (async-profiler, JFR): answers "what is the statistical picture over time?" async-profiler uses perf_events + AsyncGetCallTrace (ASGCT) to sample stacks via signals without safepoints, overhead <1%. JFR is the built-in continuous recorder with rich event library (GC, safepoints, I/O, locks), also <1% overhead, ideal as a black box.

Key insight: diagnostic tool cost differences are orders of magnitude; wrong order is more dangerous than no tool.

Command-Line Trio Best Practices

jstack: Thread Stack Snapshot

Basic usage: jstack PID. Thread states to read instantly: RUNNABLE (running or ready — not necessarily on CPU), BLOCKED (waiting for monitor), WAITING (indefinite wait), TIMED_WAITING (timed wait). Automatic deadlock detection prints "Found one Java-level deadlock". If JVM is unresponsive, jstack -F forces OS-level stack (coarser but better than nothing).

jmap: Heap Observation & Dump

jmap -histo PID

— class histogram (instance count, bytes), lightweight. jmap -histo:live PID — same but triggers Full GC first to keep only live objects.

Lightweight alternative: jcmd PID GC.class_histogram — same info, fewer pitfalls. jmap -dump:live,format=b,file=heap.hprof PID — heavy weapon: Full GC + full heap traversal, STW tens of seconds for 8 GB. Without :live includes dead objects, larger file but no extra Full GC.

jstat: GC Statistics Polling

jstat -gcutil PID 1000

prints generation usage and GC time percentages every second. Good for temporary watching. Do not put jstat in a script as a monitoring poller — each output is an attach communication that creates safepoint load. Long-term observation belongs to JFR.

jcmd: Unified Entry Point

JDK 7+ recommended: jcmd PID help lists all subcommands. Thread.print ≈ jstack, GC.class_histogram ≈ jmap -histo, GC.heap_dump, VM.flags, VM.native_memory, JFR.start/dump. New observability features appear first in jcmd; the trio can be remembered as aliases.

jstack Output Anatomy

Each thread block has five elements: name & id, state, nid (hex thread id matching top -Hp TID), stack frames (top = current method), lock info (locked / waiting on / waiting to). JVM internal threads (GC, CompilerThread, VM Thread) appearing at top are normal; but sustained high usage signals problems: GC threads busy → memory pressure; Compiler threads busy → JIT storm (deopt storm from article 08).

CPU Spikes: Classic Four-Step Method and Its Ceiling

Textbook flow for CPU 100%: top -c lock process top -Hp PID lock thread printf '%x' TID convert to hex jstack PID | grep nid=0x... -A 20 examine stack

One minute to answer "which thread burns CPU". Must become muscle memory. But three precision defects limit it to a fallback action:

Defect 1: Only stack top. The captured stack is a single instant; CPU hogs switch frames in milliseconds — single shot likely misses true hotspot.

Defect 2: Single sample. One snapshot = one frame; manual multi-shot comparison is slow and leaky.

Defect 3: No attribution. Seeing GC/JIT compiler threads RUNNABLE tells "who runs" but not "who fed it" — GC thread CPU root cause is on memory side (allocation rate), four-step method stops there.

Upgrade path: async-profiler one command: profiler.sh -d 60 -f cpu.html PID. 60-second continuous sampling produces flame graph; width = share, culprit visible instantly. Correct workflow is two-stage: four-step method for immediate triage (know where it burns), flame graph for root cause (know why it burns).

High-frequency branch: if step 2 catches a GC thread (G1* or GC task) — it's not business code burning, GC is firefighting. Switch to memory path (section 1.4) to find "who feeds it"; allocation-rate scenarios: async-profiler alloc mode more directly shows object creators. Similarly, CompilerThread burning → revisit article 08 for compilation queue & deopt storm. CPU is symptom; root cause often next door — follow thread name to switch lanes, more useful than staring at stacks till dawn.

Memory Issues: Three-Stage Rocket, Light First Then Heavy

Memory troubleshooting cost curve must be "light first, heavy later":

L1 Class Histogram (seconds) : jcmd PID GC.class_histogram lists "who owns the bulk" — no disk write, no full object graph traversal, cost = one second-level safepoint. Common suspects for memory doubling (byte[] surge, cache class instance count runaway) are obvious here; most issues end at this level.

L2 Full Heap Dump : jmap -dump:live — note live triggers Full GC first; dumping when heap nearly full risks killing service — drain traffic or histogram first.

L3 MAT Offline Analysis : Dominator Tree answers "who exclusively retains memory" — largest retained heap objects are the culprits; Path to GC Roots answers "who holds it and won't let go" — final verdict for leak attribution.

Frequent misconception correction: Class histogram only covers heap; Metaspace is not inside. Suspect Metaspace leak (CGLIB scenario from article 02) → check jcmd PID VM.metaspace or monitored Metaspace committed curve, combine with class unload logs — unload count dropping to zero means old classloader pinned. Heap leaks and Metaspace leaks are two separate investigation lines, different tools.

Overlooked histogram technique: Compare against baseline, not absolute values. A healthy service naturally has byte[]/char[] on top (caches, strings). Single histogram misleads; keep a steady-state baseline, capture second on issue, diff "which class grew" — closer to truth. Add time dimension — histogram every 10 minutes, steepest growth slope = real suspect. Memory diagnosis essence is trend reading; snapshots are just sample points for the trend.

At L3, MAT practical three steps worth rehearsing: 1) Leak Suspects report (auto suspects + suspicious reference chains), 2) Dominator Tree sorted by retained heap to find "exclusive owners", 3) Path to GC Roots on suspect objects (exclude weak/soft refs) to see what pins them — often an uncleared cache Map, unregistered listener, or static-held old classloader (article 02). Dump analysis mindset: "from large to small, from surface to depth" — same taboo as heap sizing: never start by flipping through object details.

Thread Problems: Single Snapshot vs Multi-Shot Comparison

Thread issue (hang, deadlock, lock contention) diagnosis mantra: Single shot reads state; multi-shot reads pattern. Single jstack three blind spots: transient states (just-switched BLOCKED missed), no comparison baseline (don't know what changed), static/dynamic confusion (RUNNABLE may be truly running or waiting for network ready).

Multi-shot method: 200 ms interval, 5 shots, then find two patterns:

Resident same stack (all 5 in same method = stuck; appears once = passing through)

Resident same lock (every shot has threads BLOCKED on same monitor address = lock contention hotspot)

Combine with arthas trace on suspicious method to see latency breakdown — slow layer obvious.

State quick-read adds three practical meanings: many threads BLOCKED on same lock = contention hotspot → check lock holder; many WAITING = waiting for tasks or notification → check thread pool idle or downstream; RUNNABLE but low CPU = waiting for I/O ready — typical network hang shape, exactly where section 1.6 wall mode shines.

Common hang patterns & thread signatures cheat sheet:

DB connection pool exhausted — many threads WAITING on pool's take/druid blocking queue; pool params or slow SQL root cause.

Downstream API slow — many RUNNABLE but low CPU, stack bottom socketRead; use wall flame graph to see wait share.

Lock contention — BLOCKED piled on same monitor; check holder (likely waiting for another resource; lock is just chain relay station).

Thread pool queue backlog — business threads running, submitter threads WAITING at execute entry, queue full.

Deadlock — jstack tail directly gives deadlock block, two locks mutual wait = confirmed.

Each hang investigation: match pattern first, then collect evidence — faster than blind stack flipping.

Arthas adds on thread issues: thread -b finds "busiest lock" and prints blocking threads (deadlock & contention hotspot one command); thread -n 3 lists top 3 CPU threads (built-in sampling, no manual top -Hp); thread --state BLOCKED one-click filter all lock-wait threads. It automates half of section 1.5's "multi-shot comparison" — but the other half (cross-time pattern) still needs multi-shot or profiler; command cannot replace methodology.

async-profiler: Accurate and Lightweight Principles

async-profiler's "accuracy" comes from three-layer stack:

perf_events — kernel timer interrupt sampling, only dots when target thread actually runs, no wait sampling → ultra-low overhead.

AsyncGetCallTrace (ASGCT) — JVM internal interface; in signal handler directly walks current Java stack — no safepoint, no thread pause, nanosecond return.

ASM stack walking — fills JNI & kernel native frames so flame graph shows complete call chain.

Combined effect: overhead <1%, can run continuously, samples "cumulative distribution" not single frame.

Four modes by symptom: cpu — where CPU burns. alloc — allocation hotspots: who creates objects like crazy; cross-validates with article 06 escape analysis. lock — lock wait time; incident 1's log lock contention appears as broad plateau in lock flame graph.

wall — wall-clock time sampling (includes I/O wait); network-type hangs and section 1.5 "RUNNABLE but low CPU" shape need wall to see clearly.

Flame graph reading main line: bottom-up = caller direction; wider = larger share; top wide block = culprit; "wide but short plateau" = optimization target.

Division with JFR: JFR = JDK built-in continuous recorder, rich event library (GC, safepoint, I/O, locks), suitable for always-on black box; async-profiler excels at one-off precise flame graphs. Complementary: Black box always on (JFR), incident precise locate (profiler).

Container environment two practical details: 1) perf_events needs ptrace capability; restricted K8s can fall back to itimer mode (pure user-space sampling, native frames missing but Java stack complete). 2) Flame graph's real power is in "comparison" — baseline before big promotion, another during event, generate diff flame graph; new wide plateaus = performance regression culprits. "Against baseline find increment" methodology applies equally to histogram & GC logs: isolated snapshot never as convincing as before/after comparison.

Lock mode deserves extra note: it samples "lock wait time" not "lock hold time" — so wide block in lock flame graph means "who made many threads wait long" not "who held lock longest". Holder may have done only 5 ms work, but downstream slow SQL made waiters queue for seconds — wide plateau on waiter side, root cause upstream on resource holder waits for. Reading lock flame graph: follow "waits for whom" upstream — graphical version of section 1.5 "lock is chain relay station".

Arthas: Non-Invasive Scalpel and Safety Boundaries

Arthas positioning: "online observation without restart, without code instrumentation": watch — method input/return/exception. trace — internal call latency chain. tt — time tunnel records calls for replay. sc -d — class real source & classloader (article 07 conflict troubleshooting weapon). jad — decompile runtime truth (confirm which version runs online). ognl & vmtool — fetch runtime objects as evidence.

Implementation: attach + Instrument bytecode enhancement — watch/trace essentially instrument methods with probes; probes have overhead — source of safety boundaries.

Three safety boundaries:

Watch unconditionally on high-frequency method = every call intercepts & assembles data; high QPS becomes performance amplifier — must -n limit count + condition expression filter.

Trace huge method — instruments every internal layer; deep chain latency stacks up — only trace suspicious sub-chain.

Redefine hot update bypasses release process — production forbidden — code changes go through release; diagnosis read-only. Discipline: sc -d confirm class first, drain one instance first, operate off-peak.

Arthas vs APM relationship:

APM (SkyWalking, Pinpoint, etc.) is the telescope for steady-state monitoring — panoramic metrics, trace tracking, error rate trends; Arthas is the microscope — drills into single instance to see single method, single object, single class.

APM tells you "API #5 P99 degraded"; Arthas tells you "this call's input was what, which internal layer was slow". Not competitors; sequence: APM detects anomaly → Arthas drills for evidence. Only Arthas no APM → users discover first; only APM no Arthas → localization stops at last mile.

Cleanup habit: after stop, verify enhancements truly reverted — run dashboard or empty trace to confirm no probe output; exit with quit not force kill; watch one GC cycle after attach to confirm no anomalies before closing. "Bringing probes down" as important as "bringing probes up" — many mysterious performance jitters traced to a watch left running three days ago.

Overlooked dimension: Rehearse full flow in test environment. Has cpu-doctor script run? Can profiler attach in image? Does workstation have memory for 8 GB dump MAT analysis? Can on-call read flame graphs? All zero-cost verifiable before incident; after incident all blood and tears. Put "diagnostic drill" into quarterly routine: simulate CPU spike, memory leak, hang — full flow walk-through and timed. Production composure is trained in test environment.

Production Practice: Always-On Configuration and On-Call SOP

Full Diagnostic Toolbox Config (JDK 17)

Diagnostic capability must be installed before incident — observability configured at incident time is basically none. A production-ready always-on config:

# ==== JVM Diagnostic Always-On Config (JDK 17)====
# 1. GC & Safepoint Logs: rolling, no flood
-Xlog:gc*,safepoint:file=/logs/gc.log
:time,uptime,level,tags
:filecount=5,filesize=50m

# 2. JFR Black Box Always-On (default template overhead <1%)
-XX:StartFlightRecording=maxsize=250m
# dumponexit default true, auto dump on process exit

# 3. OOM Self-Proof: auto dump once on heap full then exit
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/logs/heapdump.hprof
-XX:+ExitOnOutOfMemoryError

# 4. Class Unload Observation: normally only unload
-Xlog:class+unload=info
:file=/logs/class-unload.log

# 5. Troubleshooting-time temporary (normally OFF):
# -Xlog:class+load=info full load logs
# -Xlog:deoptimization deopt details

Five segments each keep one ledger: GC logs = bottom ledger for all JVM issues (used repeatedly in articles 03-05); JFR black box guarantees "post-incident replay of past 12 hours"; OOM auto dump lets leak scene leave own evidence — note ExitOnOutOfMemoryError lets process exit for restart, trade-off between "preserve scene" and "fast recovery", most services worth enabling; class unload logs = first signal for Metaspace leak (section 1.4); high-overhead logs for troubleshooting only, normally off, reason see section 1.2 jstat lesson.

Pre-launch checklist — tick each:

GC & safepoint logs rolling & on disk

JFR running ( jcmd PID JFR.check verify)

OOM auto dump configured & disk reserved heap-size space

class+unload logs recording

async-profiler
profiler.sh

in image & container has attach permission (/tmp shared, ptrace works)

Arthas pre-installed or one-pull script ready cpu-doctor.sh type SOP scripts in image & drill executed

Diagnostic capability essence is "muscle memory drilled from rehearsals"; tools installed but not rehearsed = still panic when incident hits.

On-Call SOP: CPU Spike Five-Minute Locate

Solidify section 1.3 method into script, zero memory burden for on-call:

#!/bin/bash
# cpu-doctor.sh: five-minute locate CPU culprit
PID=$(top -bcn1 | grep java |
head -1 | awk '{print $1}')
for i in 1 2 3; do
TID=$(top -bHp $PID -n1 |
awk 'NR>7{print $9,$1}' |
sort -rn | head -1 |
awk '{print $2}')
NID=$(printf '%x' $TID)
jstack $PID |
grep "nid=0x$NID" -A 15
>> /tmp/cpu-$(date +%s).log
sleep 2
done
# Three-shot intersection: only stacks repeating are credible
# Then profiler for root cause (two-stage):
# ./profiler.sh -d 120 -f cpu.html $PID

SOP three key points:

Three-shot intersection — single shot unreliable; repeating stack frames = real hotspot (section 1.5 multi-shot automated).

Two-stage finish — script triage must be followed by profiler command for root cause; otherwise only know thread not why.

Dump discipline — if dump needed: drain traffic first, limit to once, sensitive data (passwords, tokens in heap objects) delete after use — write into on-call handbook.

Three more scene disciplines beyond dump:

Preserve process over restart impulse — hung process is most valuable scene; before restart grab jstack, histogram, JFR dump (each seconds); only then restart with OOM auto dump fallback.

Change one variable at a time — simultaneous param tweak + scale + restart = no post-mortem attribution.

Full trace logging — every command, every timestamp into on-call log; "evidence chain" beats "memory" at retrospective.

Profiler common workflow solidified again — copy on incident:

Baseline — off-peak profiler.sh -d 60 -e cpu -f base.html PID, archive.

Incident — same command 60 sec → issue.html.

Diff — generate diff flame graph find new wide plateaus.

Chase allocation — suspect memory side switch -e alloc another round.

Chase lock — lock contention symptoms switch -e lock.

Chase wait — hang-type problems switch -e wall.

One set of five commands covers root-cause scenarios for three incident categories; instead of memorizing ten tool parameters, drill these five modes into instinct.

JFR line also not idle: jcmd PID JFR.start duration=60s filename=... record one minute; JFR.dump extract current recording; jcmd PID JFR.events list available events. Daily highest-value events: jdk.SafepointBegin/End (safepoint duration), jdk.GarbageCollection (GC details), jdk.ThreadAllocation Statistics (allocation rate), jdk.JavaMonitorEnter (lock wait). JFR advantage = structured event library — it's "queryable observation"; with JMC or jfr print can precise replay on timeline — that's black box value.

Core Source Code: How Tools See Into You

Attach Mechanism: How jstack Enters JVM

When you run jstack, it doesn't "read" target process; it gets loaded by target process. Entry in HotSpot AttachListener:

// attach: unified entry for jstack/jmap/jcmd/arthas
// Step 1: write .attach_pid<PID> file
// Location: target JVM's temp dir
// Step 2: send SIGBREAK to target process
int AttachListener::init() {
// StartAttachListener mode starts listener at process start;
default waits for signal trigger
return -1;
}

void AttachListener::handle_signal() {
// Signal received & attach_pid file exists:
// Start Attach Listener thread
JavaThread* listener = new JavaThread(
attach_listener_thread_entry);
listener->start();
}

Listener thread then listens on Unix domain socket ( /tmp/.java_pid<PID>); jstack writes commands like "thread-print", "heap-dump" into socket; JVM executes inside its own process and writes results back — this is why tools can "dialogue" with JVM, why same JDK version required for interop, and why containers must share /tmp to attach. Arthas loading uses same channel: attach then inject agent jar, Instrument mechanism starts working.

HeapDumper: Where Dump Cost Comes From

// VM_Operation: dump executes in VMThread
// i.e.: entire process at safepoint (STW)
void VM_HeapDumper::doit() {
HeapDumper::dump_heap();
// 1. Write HPROF header, class list
// 2. Traverse all oops: object+fields write one by one
// Millions of objects, I/O is main time
// 3. Write reference relations & stack maps
}
// live option: before dump schedule a full GC
// VM_Operation — only keep reachable objects
// So file smaller but wall-clock longer

This code explains incident 2 fully: dump is not a "read file" operation but a VMThread-executed full heap traversal, entire process at safepoint. Duration = heap size × object graph complexity; 8 GB tens of seconds normal; live adds a Full GC first. This is the source-code-level explanation of "heavy weapon" — and why histogram (one lightweight safepoint stats) must precede dump.

SafepointSynchronize: Why Tools All Pause the World

// Common underlying mechanism for all L1 tools
void SafepointSynchronize::begin() {
// 1. Set global poll page inaccessible
// 2. Running threads at method return/loop back-edge
// hit poll then park and wait
// 3. VMThread executes jstack/dump/histogram
// 4. end(): wake all threads continue
}
// Hang scenario: a thread runs in "uncompiled long loop"
// (08 article OSR not triggered), never hits poll
// — all threads wait with it: safepoint leak
// jstack can't capture hang, mostly this

This source strings together multiple foreshadowings: why jstack occasionally hangs no output (waiting safepoint), why dump slow (heavy work inside safepoint), why histogram also has cost (lightweight safepoint still safepoint). High-frequency interview point: Safepoint leak — a thread stuck in long loop without poll, everyone waits, symptom "random global stalls + safepoint log sync time huge" . Diagnostic means exactly this article's config -Xlog:safepoint: look at sync time not just pause time.

Real shape to build intuition: service "collectively pauses" every few minutes; single jstack captures no anomaly (world normal at capture). Open safepoint log, see Reaching safepoint duration far exceeds actual operation duration — classic "arrive slow" not "work slow". Trace up, often a scheduled task doing "uncompiled long loop" on large array or big object clone. Fixes in articles 03/08: shrink loop, add CountedLoopSafepoints so long loops can hit safepoint, or move heavy work async. Toolchain value: turns "inexplicable mystic stalls" into "has logs, has data, has fix" engineering problems.

AsyncGetCallTrace: Non-Pausing Sampling

// async-profiler core call
// SIGPROF interrupts target thread (only when running),
// signal handler directly walks Java stack:
void sample(ucontext_t* uc) {
ASGCT(frame, depth, uc);
// Read registers locate frame pointer, walk frame chain
// backtrace — pure read, no lock, no thread state change,
// nanosecond return
}
// Contrast: jstack = global safepoint for one snapshot
// profiler = every 10ms single-thread grab,
// million samples aggregate into flame graph

This is the final explanation of "accurate" and "light": sampling doesn't interrupt any thread's world; sampled thread doesn't even know its stack was read. Cost: interface doesn't guarantee every frame succeeds (threads in safepoint missed, tolerate missing ratio), and it's read-only — observation vs pause boundary is the master key to understanding entire toolchain.

Two interpretation notes for ASGCT sampling: 1) Missing rate — threads in safepoint or unparsable stacks miss samples; overall few percent normal; flame graph method share must be understood as "lower bound" not exact value. 2) Aggregation bias — short-lived threads (per-second request pool) vs long-lived (background workers) have different time-weight; thread-count-aggregated flame graph amplifies resident thread share. Understanding these two points lets flame graph numbers be "trusted but not worshipped" — it's a direction indicator, not a precision instrument.

Seven Cost Accounts: Every Tool's Real Price

Methods and source code done; now lay out the "price list". Diagnostics aren't free: each tool trades safepoints, disk, CPU, or risk for information — know these accounts to know which tool for which scene; don't know, toolbox becomes accident amplifier. Seven accounts in order:

First: jcmd GC.class_histogram ≈ one second-level safepoint. Cheapest memory observation; do first, most issues end here.

Second: jmap -dump:live ≈ Full GC + full heap traversal. 8 GB heap tens of seconds STW + 8 GB disk; heavy weapon by sequence; heap near full → drain traffic first.

Third: async-profiler always-on ≈ <1% overhead. Big-promotion escort baseline action; cpu+alloc dual-mode baseline, compare during event.

Fourth: JFR default template ≈ <1% overhead. Black box value = "post-incident replay past" — when scene destroyed by restart, only it still has evidence.

Fifth: arthas watch high-frequency unconditional ≈ one probe overhead per call. 1000 QPS method with 10 ms watch = added "observation pump"; limit count + condition filter = iron law.

Sixth: jstat 1-sec poll ≈ one attach communication per second. Troubleshooting watch 5 min fine; baked into resident monitoring = turning observation into load. Long-term → JFR or APM.

Seventh: Four-step 10 min vs flame graph 1 min. Manual multi-shot comparison ROI vs continuous sampling differs by order of magnitude — business reason for "swap shutter for video".

Interview 20 Questions (with Answers)

The article includes 20 Q&A pairs covering tool relationships, mechanisms, and practical usage. Key examples:

Q1: jstack vs jcmd Thread.print — same source/effect; jcmd is official unified entry, new features land there first; jstack older, better script compatibility. Prefer jcmd.

Q2: Why jmap -dump:live triggers Full GC — live semantics = only live objects; liveness requires full marking → implemented as Full GC VM_Operation. Side effect: longer STW, smaller more analyzable dump. Heap near full → use cautiously.

Q3: jstack occasionally hangs — waiting for safepoint. If thread in uncompiled long loop (no poll) or JVM stuck in safepoint op, all threads wait, jstack can't exit. Check safepoint log sync time, use -F for OS-level stack.

Q4: RUNNABLE ≠ using CPU — includes "ready waiting for scheduler" and network I/O ready wait (JVM doesn't distinguish). RUNNABLE but low CPU → I/O wait; confirm with wall mode sampling.

Q5: BLOCKED vs WAITING — BLOCKED waiting for monitor (synchronized entry); WAITING waiting for notification (wait/join/park, indefinite); TIMED_WAITING = timed wait. Many BLOCKED on same lock = contention hotspot; many WAITING → check thread pool & downstream.

Q6: CPU 100% standard four-step — as described; if step 2 catches GC thread → CPU burned by GC — switch to memory path find allocation hogs; don't waste time on GC thread stack.

Q7: Four-step misses hotspot → switch continuous sampling: profiler.sh -d 60 -f cpu.html PID; four-step is single frame, millisecond-switching hotspots single-shot likely miss; methodology defect not operation error.

Q8: jstack nid is hex — matches top -Hp decimal TID; hence printf %x convert; grep nid=0x -A 20 standard.

Q9: hprof file notes — disk (size ≈ heap), sensitive data (passwords/tokens in objects), retention (delete after use, don't leave in shared dir); also give MAT more memory for large dump analysis.

Q10: MAT Dominator Tree — answers "who exclusively retains memory": each object's retained heap = "how much freed if it's collected". Largest retained = leak/bloat culprits; then Path to GC Roots to see who pins them.

Q11: Class histogram replace dump? Most scenarios yes: histogram seconds gives "who owns bulk", leak bytes & instance counts obvious. Cannot replace when need reference chains (who pins it) → must dump into MAT. Light first heavy later; order cannot reverse.

Q12: async-profiler low overhead why — no instrumentation: no bytecode modification. Sampling via kernel perf_events timer interrupt + JVM internal AsyncGetCallTrace direct stack grab; signal handler nanosecond return; no safepoint, no thread pause. Fundamental difference vs Instrument (arthas watch).

Q13: alloc mode for what — allocation hotspots: who creates objects like crazy. High allocation rate amplifies GC frequency (article 03); combine with article 06 escape analysis to see "which large objects could have been scalar replaced but went to heap". Baseline before big promo, compare during event, regression nowhere to hide.

Q14: wall vs cpu mode — cpu samples only when thread actually running (excludes wait); wall samples by wall-clock time (includes I/O wait). Diagnose "stuck but not burning" (network hang, downstream wait) must use wall; diagnose "burning CPU" use cpu.

Q15: arthas watch may hang system — watch uses Instrument enhancement; every matching call executes interception logic assembling data. High-frequency method unconditional = performance amplifier: 1000 QPS method with watch = 1000 reflective value fetches + string assemblies per second. Iron laws: -n limit, condition filter, stop after observe (read-only, redefine forbidden).

Q16: Trace huge method feasible? Not recommended. Trace instruments every internal call layer; deep chain latency stacks up; huge method fills entire chain. Correct: trace to locate suspicious sub-chain, then precision trace sub-chain.

Q17: JFR vs async-profiler how choose — complementary not either/or: JFR always-on black box (rich events, historical replay); async-profiler incident precise locate (fine-grained flame graphs). Both <1% overhead; production both worth configuring.

Q18: jstat suitable scenes — temporary GC trend watch ( -gcutil per second line). Not for resident monitoring — each output = attach communication; long-term polling = manufacturing safepoint load for JVM. Long-term → JFR or APM.

Q19: OOM auto dump config & notes — -XX:+HeapDumpOnOutOfMemoryError + HeapDumpPath. Note: one OOM triggers one dump (reset after restart); add -XX:+ExitOnOutOfMemoryError for process exit to restart; balance "preserve scene" vs "fast recovery". Also HeapDumpPath disk must reserve heap-size space — OOM dump fills disk causing dump failure seen too many times.

Q20: Intermittent hang troubleshooting three axes — 1) Multi-shot compare: 200ms interval 5x jstack, find resident same stack & resident same lock; 2) arthas trace on suspicious method for latency breakdown (remember -n limit); 3) profiler wall mode for wait distribution. Core idea: intermittent problems rely on "multi-shot & continuous sampling", not single snapshot — shutter misses what camera records.

One-Page Cheat Sheet

Diagnostic Toolchain Nine Sentences:

Three-layer model: snapshot sees now, arthas sees process, profiler sees history; progressive not replacement.

All L1 tools & arthas attach go through safepoint; async-profiler uses signal no pause.

CPU four-step: top/-Hp/printf/jstack; fallback triage, root cause via flame graph.

Memory light-first: histogram → dump → MAT dominator tree.

Dump is heavy weapon: live triggers Full GC, STW proportional to heap size.

Thread issues single-shot reads state, multi-shot reads pattern; intermittent must multi-shot.

State quick-read: BLOCKED waits lock, WAITING waits notify, RUNNABLE ≠ running.

Arthas three shields: limit count, filter, read-only.

Always-on trio: GC logs, JFR black box, OOM auto dump — observability must be installed before incident.

Summary and Next Episode Preview

This article isn't a "tool list" but every observation action's "seeing mechanism" and "intervention cost" — snapshots rely on safepoints so have pauses; probes rely on bytecode enhancement so have amplification; sampling relies on signal direct-grab so near-free. Mechanism clear → order natural: always-on first (GC logs, JFR, OOM dump), light always first (histogram before dump, four-step before profiler), heavy weapons with discipline (drain traffic, limit once, delete after use), online observation with bottom line (limit, filter, read-only).

Learning path: first drill four-step into muscle memory (one real incident beats ten tutorials); then in test env run profiler five modes each capture a flame graph to build "reading" feel; then practice arthas watch/trace/sc three pieces; finally re-read this article's Attach, dump, safepoint, ASGCT four source sections — all tool behaviors suddenly make sense. Tools become obsolete (jstat will be replaced by JFR), but "seeing mechanism × intervention cost" analytical framework won't.

Review three incidents: uncatchable culprit → answer continuous sampling not twenty more shots; dump-dragged afternoon → answer histogram first not reflex heavy weapon; single-shot invisible hang → answer 200ms multi-shot compare not "looks normal". Complete diagnostic methodology in one sentence: observability installed before incident (always-on config), cost light-first (histogram before dump), intermittent relies on multi-shot & sampling (shutter swapped for video) — tools have no superiority, wrong order is the accident.

Next episode "Production Practice": consolidates all weapons from previous nine articles into a landing-ready JVM production parameter full skeleton, pre-launch checkup script, ten-category incident quick-reference, and big-promo contingency plan — JVM series 10-episode finale, putting every parameter you've learned onto one battle map.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

JVMSafepointDiagnosticsperformance analysisarthasmemory leaksasync-profilerjcmdjstackjmapJFRflame graphsthread dumps
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.