Java Server Troubleshooting: 5 Essential Linux Commands for Root-Cause Analysis

Introduction: Shifting from JVM to Server-Level Troubleshooting

This article is the sixth in a series on Java online issue investigation and performance tuning. Previous parts covered troubleshooting mindset, logging systems, JDK diagnostic commands, Arthas live debugging, and GC log tuning—all focused on the Java application and JVM layers. However, real production failures are layered: application problems are often symptoms, while server resource exhaustion (CPU saturation, memory exhaustion, disk full, network packet loss, port exhaustion, connection pool depletion) is the underlying root cause. Developers who lack server command skills remain stuck at code-level debugging, unable to locate bottom-layer resource issues, leading to recurring failures and incomplete investigations.

The article introduces a four-layer troubleshooting logic that should be applied from bottom to top:

Server Resource Layer (CPU/Memory/Disk/Network) → JVM Layer (GC/Threads/Memory) → Middleware Layer → Business Code Layer

Most developers reverse this order, wasting time on code and logs first. True production troubleshooting always starts by confirming server health.

1. CPU Troubleshooting: The top Command

Basic Usage and Core Metrics

Run top for a real-time view of process CPU usage, memory usage, and load status—the primary entry point for investigating service latency and CPU spikes.

Three critical metrics for production diagnosis:

load average : 1-minute, 5-minute, and 15-minute system load. Load approaching the number of CPU cores indicates full utilization; load far exceeding core count signals severe overload.

%CPU : Per-process CPU usage. Sustained 100% on a single process means that process is CPU-bound.

%MEM : Per-process memory usage, useful for spotting resident memory that isn't released.

Java-Specific Diagnostic Workflow (Production Standard)

Use top to identify the Java process with the highest CPU consumption (PID).

Run top -H -p PID to list all threads within that process and pinpoint the thread (TID) consuming excessive CPU.

Convert the TID to hexadecimal with printf, then use jstack to locate the exact code line causing the high CPU.

Combine with Arthas trace command to verify infinite loops, deadlocks, or logic delays.

Anomaly Thresholds

Single Java process CPU > 80% continuously: indicates code infinite loops, heavy computation, or frequent GC.

Overall server CPU > 90% continuously: machine resource bottleneck; requires scaling, rate limiting, or task optimization.

Sustained high load: thread pile-up, request blocking, severe task backlog.

2. Memory Troubleshooting: The free Command

Common Production Command

# Human-readable memory units (GB/MB)
free -h

Core Field Interpretation (Key Pitfall Avoidance)

total : Total physical memory on the server.

used : Memory currently used (processes + cache + buffers).

free : Completely idle memory.

available : Truly usable memory (the only field to watch in production).

The biggest novice mistake is looking only at free and ignoring cache usage, leading to false "out of memory" conclusions. In production, judge memory sufficiency solely by available.

High-Frequency Failure Scenarios

available continuously dropping : Server has a memory leak or processes holding resident memory without release.

Cache usage too high : Frequent disk I/O or massive file reads prevent the system cache from auto-reclaiming.

Memory repeatedly hitting 100% : Triggers the OOM killer, which forcibly terminates the Java process, causing unexplained service restarts.

3. Disk Troubleshooting: The df Command

Core Diagnostic Commands

# Overall disk usage
df -h

# Directory-level file sizes to locate large files
du -sh *

Top 3 Root Causes of Disk Full (99% of Cases)

Logs not rotated or cleaned : Unbounded log accumulation consuming tens of gigabytes.

Error log flooding : Code infinite loops generating massive error logs in a short time.

Temporary files and heap snapshots not cleaned : jmap -generated hprof files and temporary cache files occupying disk.

Production Standards and Remediation

Disk usage > 80% must trigger alerts; > 90% requires emergency cleanup. Daily operations must enforce log rotation and scheduled archival cleanup to prevent disk-full incidents.

4. Network Troubleshooting Part 1: The netstat Command

High-Frequency Production Commands

# View all TCP connections, ports, and connection states
netstat -an

# Count current established connections
netstat -an | grep ESTABLISHED | wc -l

Critical Connection State Interpretation

LISTEN : Port listening normally; service started successfully.

ESTABLISHED : Normal active connections; excessive count indicates connection pile-up.

Excessive TIME_WAIT : Frequent short-lived connection creation/destruction exhausts port resources, blocking new requests.

Excessive CLOSE_WAIT : High-frequency fatal issue in production! Client closed connection, but server side did not release it, causing connection leaks, connection pool exhaustion, and service hangs.

CLOSE_WAIT Root Cause and Resolution

Large numbers of CLOSE_WAIT are a classic Java network fault. Root causes: code failing to close IO streams, not releasing connections, or not handling external interface timeouts, leaving connections hanging indefinitely.

Diagnostic approach: Combine Arthas trace to track external calls and IO operations, fix resource leaks, and optimize timeout mechanisms.

5. Network Troubleshooting Part 2: The tcpdump Command

Common Production Capture Commands

# Capture all packets on a specific port
tcpdump -i any port 8080

# Capture packets for specific IP + port, save to file for later analysis
tcpdump -i any port 8080 and host 192.168.1.100 -w log.pcap

Practical Application Scenarios

Intermittent interface timeouts: determine whether the service never received the request or received it but failed to respond.

Missing or truncated request parameters: identify whether the issue originates from the frontend, gateway, or the service itself.

Cross-server call failures: examine packet loss, retransmissions, or connection refusals.

Note: Production packet captures should be short-lived and targeted to avoid impacting server performance.

6. Universal Four-Layer Server Troubleshooting Workflow (Production-Ready)

Integrating all commands into a repeatable 6-step process for any service anomaly:

Check CPU : top to view load and CPU usage, identify process/thread bottlenecks.

Check Memory : free -h to confirm machine memory adequacy and detect memory buildup.

Check Disk : df -h to verify disk isn't full and logs aren't accumulating abnormally.

Check Network : netstat to investigate connection pile-up, port anomalies, and connection leaks.

Deep Packet Capture : tcpdump to pinpoint intermittent network timeouts and packet loss.

Layer Upward : Only after confirming server health, proceed to troubleshoot JVM, middleware, and business code.

7. Summary

Server commands form the foundational bedrock of online troubleshooting. All Java application faults ultimately rely on server state for final verdict. This article thoroughly covers the five production-critical commands— top for CPU, free for memory, df for disk, netstat for connections, and tcpdump for packet capture—closing the gap in bottom-layer resource diagnostics and enabling a complete troubleshooting chain from server → JVM → business code. Developers gain full-stack fault location capability , no longer limited to the code layer.

8. Next Episode Preview

The next installment dives into JVM high-frequency fatal fault practice: a hands-on, end-to-end postmortem of a CPU 100% spike —from symptom reproduction and thread analysis to root cause identification, solution implementation, and long-term preventive measures—covering the most common production incident and a must-know interview topic.