10 Essential Linux Commands to Quickly Diagnose 80% of Production Issues

1. Secure the incident before running commands

Record alert time, hostname, service instance, recent changes, monitoring window, and current owner. Verify you are on the correct host with hostnamectl and date --iso-8601=seconds. All commands are read‑only samples; avoid installing packages on production during a crisis.

2. Recommended command order

Start with uptime to see load and time, then top for current hotspots, followed by vmstat to differentiate resource types, mpstat to tell single‑core vs. whole‑machine saturation, pidstat to pinpoint offending processes/threads, iostat for block‑device bottlenecks, ss for socket state, df for filesystem capacity and inode usage, journalctl to align events with logs, and finally dmesg for kernel, OOM, and device errors.

Command 1: uptime – confirm load trend and fault time

uptime

Outputs current time, uptime, user count, and 1‑, 5‑, 15‑minute load averages. A rising short‑term load suggests recent pressure but does not prove CPU saturation. Compare load against logical CPU count; high load with many idle CPUs often indicates I/O or kernel wait.

Command 2: top – quickly lock in resource hotspots

LC_ALL=C top -b -d 1 -n 10 > /tmp/top-$(date +%Y%m%d-%H%M%S).txt

Collect ten one‑second snapshots. Before sampling, check df -h /tmp and df -i /tmp to ensure enough space and inodes. In the header, focus on load average, task states, CPU time, memory, and swap. In the process list, watch PID, user, state, %CPU, %MEM, runtime, and command. The %CPU denominator includes all cores, so a multithreaded process can exceed 100%. Press H to toggle thread view or target a specific PID:

PID=12345
top -H -p "$PID"

Multiple samples are required; a single frame is insufficient to identify root cause.

Command 3: vmstat – assess run queue, paging, and overall system state

vmstat 1 10

First line shows averages since boot; focus on subsequent lines. Key fields: r: runnable tasks – high values above CPU count indicate queueing. b: uninterruptible sleep – rising values often mean I/O or kernel wait. si / so: swap in/out – activity matters more than mere swap usage. in / cs: interrupts and context switches – compare against baseline. us, sy, id, wa, st: CPU time distribution.

Typical useful combinations: high r + high us + low id points to CPU‑bound work; high b + high wa suggests storage or uninterruptible I/O issues; high st indicates virtualization steal time.

Command 4: mpstat – distinguish single‑core saturation from whole‑machine load

mpstat -P ALL 1 10

Shows per‑CPU percentages. Example output reveals CPU 0 near 100 % while CPU 1 remains idle, hinting at a single‑thread hotspot, CPU affinity, or scheduling issue rather than a need for horizontal scaling.

Command 5: pidstat – map resource anomalies to processes and threads

pidstat -u -r -d -w 1 10

Aggregates CPU, page faults, I/O, and context switches. To focus on a specific PID:

PID=12345
pidstat -t -u -r -d -w -p "$PID" 1 10

Do not rely on a single %CPU ranking; high CPU may be normal traffic. Examine process start time, full command line, and, for Java, use jcmd PID Thread.print -l (after confirming low overhead). Sensitive parameters must be redacted before sharing diagnostics.

Command 6: iostat – determine if block devices are bottlenecks

iostat -xz 1 10

Shows extended stats; -z hides idle devices. Key fields include read/write IOPS, throughput, average request size, queue length, await times, and %util . For traditional single‑queue disks, %util near 100 % signals saturation; for NVMe or multi‑queue devices, high %util alone is not conclusive. Correlate with kernel logs and storage metrics.

Command 7: ss – inspect listening sockets, connection states, and queues

ss -lntp

Lists listening ports and owning processes. Summarize TCP states with ss -s . To check a specific service port:

PORT=443
ss -ntp "sport = :$PORT"

Non‑root users may not see all processes. Interpret Recv-Q and Send-Q according to socket role; high SYN-RECV can indicate attacks or accept‑path bottlenecks, while many TIME-WAIT entries are normal but need context.

Command 8: df – verify filesystem capacity, inode usage, and mount points

df -hT
df -i

Disk full can be space‑exhaustion or inode‑exhaustion. Use findmnt -T /var/log to confirm the mount point. If df and directory size differ, check for deleted but still‑open files with lsof +L1 . Deleting files without proper verification can cause data loss.

Command 9: journalctl – align resource observations with service events

SERVICE=example.service
journalctl -u "$SERVICE" --since '-30 min' --no-pager
journalctl -u "$SERVICE" --since '2026-07-10 14:00:00' --until '2026-07-10 14:30:00' -p err --no-pager -o short-iso-precise

Show recent error‑level logs, but also review warnings and info for context. Correlate with ss , latency metrics, and upstream logs. Be aware of log rotation and possible missing entries.

Command 10: dmesg – confirm kernel, OOM, device, and filesystem anomalies

dmesg -T --level=emerg,alert,crit,err,warn

Human‑readable timestamps may drift; for precise correlation use journalctl -k -o short-iso-precise . Look for OOM/killed processes, block device timeouts, filesystem read‑only switches, NIC link changes, soft lockups, or RCU stalls. Kernel log access is often gated by kernel.dmesg_restrict ; avoid lowering security just to read logs.

Putting the ten commands together

When an alert shows interface timeout and rising node load, follow this loop:

Run uptime to confirm load increase.

Run top to see if the load is CPU‑bound or I/O‑blocked.

Run vmstat and examine r, b, wa, si/so.

Run mpstat -P ALL to locate per‑CPU pressure.

Run pidstat to find the offending process/thread.

Run iostat to check device queues and latency.

Run ss to verify listening sockets and connection queues.

Run df -hT and df -i to ensure storage is not exhausted.

Run journalctl to align service restarts, retries, or timeouts.

Run dmesg (or journalctl -k) for kernel‑level clues.

Each step should produce a decision: continue probing, gather more evidence, or move to remediation.

Common misjudgments

High load does not always mean CPU shortage; combine with mpstat and vmstat to differentiate.

Low free memory is not necessarily a leak; examine MemAvailable, per‑process RSS, cgroup limits, PSI, and swap activity.

Port unreachable is not always a firewall issue; first verify the service is listening with ss -lntp before changing network rules.

Evidence required before remediation

Before deleting data, restarting services, or changing kernel parameters, collect:

Impact scope (hosts, instances, downstream services).

Pre‑change health checks (capacity, error rates, latency).

Backup of configuration and relevant logs.

Dry‑run or gray‑scale validation.

Verification plan covering metrics, error rates, and business functionality.

Rollback procedure with clear thresholds.

Post‑fix validation and post‑mortem

Re‑run the same command set after the fix and confirm that load, run queue, CPU types, paging, I/O wait, process metrics, disk utilization, socket state, and error rates have returned to baseline. Document the original commands, timestamps, key outputs, change timeline, root cause, remediation steps, and rollback outcome to aid future incident responders.