Java Production Troubleshooting: Universal SOP & 10 Failure Cheat Sheet

Core Top-Level Thinking: Unified Layered Diagnosis Model

All Java production faults, no matter how bizarre or complex, follow the core principle of bottom-up, layer-by-layer elimination — a foundational mindset for senior engineers that applies for life:

Server Resource Layer → JVM Virtual Machine Layer → Middleware Layer → Database Layer → Business Code Layer

Absolute taboo : Novices troubleshoot top-down (code first, logs first); senior engineers always go bottom-up (exclude underlying resource bottlenecks first). If the bottom layer is abnormal, upper-layer code analysis is useless.

Layer Fault Core Characteristics Quick Reference

Server Layer Faults : CPU/memory/disk/network anomalies; global service sluggishness, all interfaces affected.

JVM Layer Faults : GC anomalies, thread anomalies, memory leaks; service jitter, occasional timeouts, gradual degradation.

Database Layer Faults : Slow SQL, lock waits, large transactions; slow interfaces, peak-hour avalanches, single business module anomalies.

Code Layer Faults : Logic bugs, null pointers, infinite loops; fixed errors, fixed scenarios, stable reproduction.

Universal Production Fault Troubleshooting SOP (6-Step Method)

Integrating the entire series, this universal 6-step method applies to any production fault without ad-hoc thinking.

Core Principle: Stop the bleeding first, collect evidence second, root-cause fix last — never blindly restart and lose the scene.

Step 1: Confirm Fault Phenomenon & Assess Impact Scope

First moment of fault outbreak — confirm the problem, forbid blind operations:

Confirm fault manifestation: interface timeout? service hang? CPU spike? log errors? page anomalies?

Confirm impact scope: global service, single node, single module, single interface?

Confirm fault period: off-peak sudden, peak recurrence, worsens over runtime?

Step 2: Underlying Resource Quick Bottom-Line Check (30-Second Bottleneck Location)

Prioritize executing five core server commands to quickly exclude underlying issues: top: Check CPU load, process usage, exclude CPU saturation bottleneck. free -h: Check physical memory, available memory, exclude memory exhaustion, OOM killer. df -h: Check disk usage, exclude disk full, logs unwritable. netstat: Check TCP connections, CLOSE_WAIT, connection pile-up, exclude network connection leaks. dmesg: Check system kernel logs, confirm if process was killed by system.

Step 3: JVM Layer Deep Diagnosis (Java Service Exclusive Core)

Server resources normal → 100% JVM and application layer issues, targeted diagnosis:

GC Diagnosis : jstat -gc observe GC frequency, GC duration, FGC count, judge if GC explosion.

Thread Diagnosis : jstack /Arthas thread view thread states, locate deadlock, thread pool exhaustion, blocked threads.

Memory Diagnosis : jmap /MAT analyze heap snapshots, locate large objects, memory leaks, resident objects.

Real-time Diagnosis : Arthas trace / watch trace time-consuming methods, exception chains.

Step 4: Database & Middleware Layer Diagnosis

JVM normal but interfaces still slow/timeout → lock data layer bottlenecks:

Check slow query logs, capture timeout SQL. EXPLAIN analyze execution plan, troubleshoot index failure, full table scans.

Check long transactions, lock waits, deadlock blocking.

Troubleshoot Redis, MQ, cache penetration/breakdown/expiration pile-up issues.

Step 5: Emergency Stop-Bleeding, Rapid Business Recovery

After locating rough bottleneck, prioritize business availability, then deep-dive root cause:

Resource bottleneck: scale nodes, switch traffic, degrade non-core business.

Thread/memory leak: export snapshot for evidence then restart service.

Slow SQL bottleneck: temporary SQL optimization, add index, kill blocking sessions.

Code bug: temporary rollback version, take down problematic interface.

Step 6: Root Cause Location, Fix Implementation, Postmortem Prevention

Business recovered → deep postmortem to eradicate, prevent recurrence:

Based on logs, stack traces, snapshots, monitoring, precisely locate code root cause.

Code fix, parameter tuning, architecture optimization.

Add monitoring alerts, complete patrol mechanisms, improve code standards.

Output fault postmortem document, precipitate team pitfall-avoidance experience.

Top 10 High-Frequency Production Faults Quick-Reference Manual (Direct Lookup at Work)

All practical faults from the series condensed into Phenomenon-Root Cause-Solution cheat sheets for direct matching during incidents.

1. CPU 100% Spike

Core Phenomenon : High load, CPU saturated, large-area interface timeouts, no log errors.

Core Root Cause : Code infinite loops, frequent Full GC, massive spin locks, intensive computation.

Diagnostic Tools : top → top -H → jstack → locate code line.

2. Service Hang, Zero Interface Response

Core Phenomenon : Resources normal, service alive, all interfaces blocked with no return.

Core Root Cause : Thread deadlock, thread pool exhaustion, massive WAITING/BLOCKED threads.

Diagnostic Tools : jstack check deadlock, count thread states, Arthas trace blocking tasks.

3. Service Slows Over Days, Eventually OOM Crash

Core Phenomenon : Normal start, runs days then lags, old generation only grows, auto-restart.

Core Root Cause : Memory leak (ThreadLocal/static collections/unreleased connections).

Diagnostic Tools : jmap, MAT heap snapshot analysis, continuous GC log observation.

4. Peak-Hour Batch Interface Timeouts

Core Phenomenon : Off-peak normal, peak avalanche, thread pile-up.

Core Root Cause : Slow SQL blocking, external interfaces without timeout, unreasonable thread pool parameters.

Diagnostic Tools : Slow query logs, EXPLAIN, Arthas trace time-consuming chains.

5. Disk Full Service Abnormal

Core Phenomenon : Logs unwritable, service errors, startup anomalies.

Core Root Cause : Log flooding, large file accumulation, no log rotation strategy.

Diagnostic Tools : df -h, du -sh locate large files.

6. Service Mysterious Auto-Restart

Core Phenomenon : No manual operation, service timed/random restarts.

Core Root Cause : Physical memory exhausted → system OOM-kill, JVM crash.

Diagnostic Tools : dmesg, system logs, OOM snapshots.

7. Occasional Interface Jitter, Response Unpredictably Fast/Slow

Core Phenomenon : Mostly normal, occasional sudden latency spikes.

Core Root Cause : Frequent Minor GC, cache invalidation, instantaneous large objects, network fluctuations.

Diagnostic Tools : GC logs, interface monitoring, cache monitoring.

8. Database CPU Surge

Core Phenomenon : Application normal, database load maxed, all queries sluggish.

Core Root Cause : Index failure, full table scans, unoptimized large SQL.

Diagnostic Tools : Slow query logs, EXPLAIN execution plan.

9. Transaction Timeout, Data Update Failure

Core Phenomenon : Data commit fails, transaction rollback, occasional deadlock errors.

Core Root Cause : Long transactions, slow SQL holding row locks, lock wait pile-up.

Diagnostic Tools : innodb_trx, lock status queries, transaction duration monitoring.

10. No Errors in Logs But Business Anomaly

Core Phenomenon : No ERROR logs, no stack traces, business data abnormal.

Core Root Cause : Network packet loss, cache anomalies, silent thread blocking, hidden parameter anomalies.

Diagnostic Tools : tcpdump packet capture, Arthas real-time observe method input/output parameters.

Golden Principles of Production Fault Handling (Career Pitfall Avoidance)

Years of fault postmortems distilled into 5 iron laws of production faults , lifelong value:

Fault Priority: Availability First : Business first, stop bleeding and recover first, then discuss root cause optimization.

Never Blindly Restart : Restart loses all fault scene, small problem drags into intractable disease.

No Monitoring, No Production : All production hidden dangers ultimately rely on monitoring for early interception.

Slow Problems More Dangerous Than Fast Ones : Memory leaks, slow GC, accumulated hidden dangers — outbreak equals major accident.

All Occasional Problems Have Root Causes : No inexplicable faults, only hidden bugs you haven't found.

Full Series System Summary: From Entry to Senior Capability Transformation

The complete 11-article series achieves Java developers' troubleshooting capability transformation :

Junior Developer : Only reads business logs, only restarts service, helpless against complex faults.

Mid-Level Developer : Proficient with JDK commands, Arthas, can troubleshoot CPU, thread, memory, GC and other JVM faults.

Senior Developer/Architect : Possesses full-chain diagnosis capability , from server bottom layer, JVM, middleware, database, code layer all-round problem location, handles difficult hidden faults, can pre-optimize and prevent risks, guarantees high system availability.

Production troubleshooting ability is the salary watershed and core workplace competitiveness for Java engineers, and the core hard skill for interviews, promotions, and architecture implementation.

Closing Remarks

The "Java Production Fault Full-Chain Troubleshooting Practical Series" officially concludes here.

From tool usage to practical cases, from single-point faults to systematic methodology, from passive firefighting to active prevention, the entire content covers all core scenarios for work, interviews, and production operations.

Hope everyone not only collects and memorizes, but also lands in practice and applies learning , truly converting troubleshooting ability into personal workplace hard skill, bidding farewell to production fault anxiety, calmly handling all production issues.

May every production alert find you calmly diagnosing, precisely locating, and confidently resolving!