10 Linux Commands to Diagnose 80% of Production Incidents

Before Running Commands: Preserve the Incident Scene

Record alert time, hostname, business instance, recent changes, monitoring time range, and current handler. Confirm you are logged into the correct environment:

hostnamectl
date --iso-8601=seconds

Commands in this article are primarily read-only sampling, but read-only does not mean zero overhead. Short-interval sampling, traversing many processes, or reading massive logs can increase CPU, I/O, or log-system pressure. Start with lightweight commands, limit count and time range; do not copy example hostnames, PIDs, device names, or timestamps directly into production.

Recommended sequence: use uptime to judge time and load, top to see current hotspots, then vmstat and mpstat to differentiate resource types; next pidstat to pinpoint processes, iostat for block devices, ss for network connections; finally check filesystem capacity, service logs, and kernel logs.

Command 1: uptime — Confirm Load Trend and Incident Time

uptime

Example output:

15:20:01 up 42 days, 3:18, 2 users, load average: 12.40, 10.85, 6.20

uptime

shows current time, uptime, logged-in users, and 1/5/15-minute load averages. In the example, short-term load exceeds long-term, indicating recent load increase, but this does not yet prove CPU saturation.

Linux load average reflects runnable plus uninterruptible-sleep tasks. CPU-intensive work raises load, but disk, NFS, or other kernel waits can also raise it. You must know logical CPU count:

nproc
lscpu

Sustained load significantly above logical CPU count usually indicates queueing, but no single multiplier fits all systems. Service type, CPU topology, task states, and historical baselines all affect judgment. Low load does not prove health: network failures, service exits, or absent traffic can leave the system idle.

If uptime shows a recent reboot, immediately correlate with changes and boot logs. Short uptime is factual evidence, not proof of abnormal restart; verify with journalctl --list-boots and infrastructure records.

Command 2: top — Quickly Lock Current Resource Hotspots

Interactive view: top Collect 10 samples at 1-second intervals and save the scene:

LC_ALL=C top -b -d 1 -n 10 > /tmp/top-$(date +%Y%m%d-%H%M%S).txt

Check /tmp space and inodes before writing to avoid worsening disk issues:

df -h /tmp
df -i /tmp

top

header focuses on load average, task states, CPU time, memory, and swap; process list on PID, user, state, CPU%, memory%, runtime, and command. Common CPU fields: us (user), sy (system), id (idle), wa (I/O wait), st (steal). Field names and display modes depend on procps-ng version and interactive config.

Process %CPU denominator matters: on multi-core hosts a single multi-threaded process can exceed 100%. This is not a monitoring error. top 's interactive toggles also affect display; record logical CPU count and tool mode in analysis reports.

Press H to toggle thread view, or target a specific PID:

PID=12345
top -H -p "$PID"

PID=12345

is a placeholder; replace with confirmed real PID. top represents only the sampling instant; do not conclude root cause from a single frame. Observe multiple sampling cycles and align with alert time, request volume, and logs.

Command 3: vmstat — Judge Run Queue, Paging, and Overall System State

vmstat 1 10

Example output:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
10  0      0 642000  18000 820000    0    0    10    40 6200 18000 78 18  4  0  0

The first line is typically the average since boot; real-time analysis focuses on subsequent lines. r: runnable tasks. Persistently higher than logical CPUs indicates run-queue contention. b: uninterruptible-sleep tasks. Sustained rise often points to I/O or kernel waits, but needs further forensics. si, so: swap-in/out per second. Swap usage alone doesn't mean active paging; watch for sustained activity during sampling. bi, bo: block device read/write rates (units per local version). in, cs: interrupts and context switches per second. Absolute values need comparison with same-host historical baselines. us, sy, id, wa, st: CPU time distribution.

Typical combinations outweigh single fields: r high + us high + id low → business compute hotspot; b and wa both high → storage or uninterruptible wait; st high on VMs → correlate with host-platform metrics; si / so persistently active with rising latency → memory pressure amplifying via paging. vmstat memory fields cannot replace /proc/meminfo or cgroup stats. When a container hits its memory limit, host free may still show plenty.

Command 4: mpstat — Distinguish Single-Core Saturation from Whole-Machine Saturation

mpstat -P ALL 1 10

Example output:

Average:     CPU   %usr  %nice   %sys %iowait  %irq %soft %steal %idle
Average:       0  96.00   0.00   3.00    0.00  0.00  0.50   0.00  0.50
Average:       1  18.00   0.00   4.00    1.00  0.00  1.00   0.00 76.00

Example shows CPU 0 near saturation while CPU 1 has headroom. This may stem from a single-threaded hotspot, CPU affinity, softirq distribution, or application scheduling—further process/thread inspection needed; simply scaling the whole machine may not help.

Key judgments: %usr high: typical of business compute, compression, encryption, serialization, garbage collection. %sys high: check syscalls, locks, network packet processing, frequent context switches, or kernel paths. %soft high: combine with /proc/softirqs, NIC packet rates, drops, and interrupt affinity. %iowait high: CPU idle with pending block I/O; not equivalent to disk utilization. %steal high: vCPU waiting for host scheduler; requires cloud/virtualization platform evidence.

Looking only at all-CPU average masks single-core bottlenecks. Conversely, a brief 100% on one CPU doesn't necessarily cause business failure; correlate with thread hotspots and request latency.

Command 5: pidstat — Attribute Resource Anomalies to Processes and Threads

Combined sampling of CPU, page faults, I/O, and context switches: pidstat -u -r -d -w 1 10 Watch only a specific process and its threads:

PID=12345
pidstat -t -u -r -d -w -p "$PID" 1 10

pidstat

answers: "During whole-machine anomaly, which process persistently consumes resources?" -u CPU, -r page faults/memory, -d I/O, -w context switches, -t show threads. Parameter combinability and fields vary by sysstat version; check pidstat --help or man page first.

Do not judge root cause by a single %CPU ranking. A high-CPU process may be handling normal traffic; the real anomaly could be traffic surge or downstream timeout retries. Also check process start time and full command line:

ps -p "$PID" -o pid,ppid,user,stat,psr,pcpu,pmem,rss,vsz,etime,lstart,cmd
cat "/proc/$PID/status"
cat "/proc/$PID/limits"

Command-line arguments may contain passwords, tokens, or connection strings; sanitize before saving or sharing diagnostic files. Process RSS may include shared pages; summing RSS across processes overstates total memory. For precise analysis examine /proc/$PID/smaps_rollup, but this data is lost after process exit.

If a thread shows sustained high CPU, use the corresponding runtime's thread dump or sampling tool. For Java: jcmd PID Thread.print -l. Attaching diagnostic tools incurs brief overhead and output may contain sensitive class names/paths; evaluate on a single instance first.

Command 6: iostat — Determine If Block Devices Are Bottlenecks

iostat -xz 1 10

-x

shows extended stats; -z hides devices with no activity during sampling. The first device report may be the since-boot average; real-time analysis focuses on subsequent intervals.

Common fields: read/write IOPS, throughput, average request size, queue, read/write await, and device busy percentage. sysstat versions differ; you may see r_await, w_await, aqu-sz, avgqu-sz, %util, etc.

Judgment cannot rely solely on %util:

For traditional single-queue disks, %util near 100% often indicates device busy.

For NVMe, multi-queue devices, RAID, SAN, and cloud disks, parallel request handling means %util alone cannot represent saturation. await is average time from queue to completion; must be judged with device type, business SLO, IOPS, throughput, and queue together.

Application write slowness may originate from filesystem, log sync, storage network, or remote service—not solely local disk.

Further pinpoint processes with: pidstat -d 1 10 If iostat shows rising latency and kernel logs reveal timeouts, resets, or filesystem errors, evidence strengthens. Do not run cache drops, unmount filesystems, or reboot nodes first—these may expand impact and destroy the scene.

Command 7: ss — Check Listeners, Connection States, and Queues

View listening ports and processes: ss -lntp View TCP state summary: ss -s View TCP connections for a target port (replace with real service port):

PORT=443
ss -ntp "sport = :$PORT"

Non-root users may not see other users' process info. ss helps confirm whether service is listening, on which address, current connection states, and socket queues. Recv-Q / Send-Q on listening lines have different semantics than on established connections; do not uniformly interpret as "network backlog." Listening socket queue info assists accept-queue judgment; established connections reflect receive/send queues—exact semantics per ss(8) and kernel implementation.

When summarizing connection states, avoid unrestricted complex pipelines scanning all connections at high frequency; on hosts with huge connection counts this adds overhead. Use ss -s for summary first, then narrow by port and address.

Common judgments:

No listener: check service status, bind address, startup logs, and config—not firewall first.

Large SYN-RECV growth: possible burst connections, upstream retries, attack, or accept-path bottleneck; correlate with packet rates, SYN cookies, load balancer, and app processing capacity.

Many TIME-WAIT: normal TCP state; evaluate with new-connection rate, port range, connection reuse, and business impact—do not blindly clean.

Persistent CLOSE-WAIT accumulation: usually indicates local application not closing sockets after remote close; locate process and code path.

Modifying firewall, kernel TCP params, or conntrack config are high-risk operations. Without packet captures, counters, and path evidence, do not mask application issues by relaxing parameters.

Command 8: df — Check Capacity, Inodes, and Mount Points

View filesystem capacity and type: df -hT View inodes: df -i Disk "full" has two common forms: space exhaustion and inode exhaustion. Many small files can drive df -i to 100% while df -h still shows space; apps then fail to create files, write logs, or unpack deployments.

Confirm which mount point a path actually resides on: findmnt -T /var/log When df and directory stats disagree, check for deleted but still-open files:

lsof +L1

lsof +L1

may traverse many objects on large hosts; during incidents limit by user, process, or path. Deleted files held open by processes do not release space. Do not restart processes immediately; first confirm which service owns the file, whether it can safely reopen logs, and if unwritten data risks loss.

Disk cleanup is high-risk. Impact may include application data, audit logs, database WAL/binlog, container layers, and system boot. Before execution: confirm file purpose, holding process, retention policy, backup, and recovery method; generate candidate list and manually review; use application or log-system official rotation mechanisms for gradual handling. Never run unrestricted rm -rf, find ... -delete, or truncate commands directly on production directories.

Command 9: journalctl — Align Resource Symptoms with Service Events

View last 30 minutes of a specific service:

SERVICE=example.service
journalctl -u "$SERVICE" --since '-30 min' --no-pager

SERVICE

is a placeholder; replace with real systemd unit name. View error-priority logs with precise timestamps:

journalctl -u "$SERVICE" --since '2026-07-10 14:00:00' --until '2026-07-10 14:30:00' -p err --no-pager -o short-iso-precise

Dates are examples; replace with alert window. -p err shows only error and higher; during troubleshooting do not look only at errors—critical context may be logged as warning or info. Suggest first reading full context within time and unit, then keyword filtering.

View service status and current-boot logs:

systemctl status "$SERVICE" --no-pager
journalctl -u "$SERVICE" -b --no-pager

Focus alignment on: process start/exit, config load, dependency connection failures, timeouts, retries, OOM, health checks, and deploy operations. A single log anomaly ≠ root cause. Example: "database connection timeout" could stem from DB failure, network loss, pool exhaustion, or app thread blocking; must cross-validate with ss, latency metrics, and downstream logs.

journald may not have persistence enabled; logs may have rotated. Absence of logs only means current data source didn't record it, not that the event didn't happen. When reading/exporting logs, watch for credentials, user data, and request content sanitization.

Command 10: dmesg — Confirm Kernel, OOM, Device, and Filesystem Anomalies

Read kernel ring buffer with human-readable timestamps:

dmesg -T --level=emerg,alert,crit,err,warn

dmesg -T

timestamps may be affected by system suspend/resume or clock changes. For precise alignment with log platform, prefer:

journalctl -k --since '-2 hours' --no-pager -o short-iso-precise

Common items to watch: Out of memory, oom-kill, Killed process: confirm kernel OOM or memory cgroup OOM.

Block device timeout, reset, I/O error: combine with iostat and storage-side metrics.

Filesystem read-only, journal error, metadata error: immediately assess data consistency and write impact.

NIC link down/up, driver reset, queue anomalies: correlate with interface stats and switch/cloud logs.

Soft lockup, hung task, RCU stall: may involve CPU stall, kernel wait, or driver issues; preserve full context and kernel version.

Kernel log access is typically controlled by kernel.dmesg_restrict. Do not temporarily lower security parameters for convenience; obtain via authorized accounts or established log platforms. After seeing hardware or filesystem errors, do not reboot directly—reboot may temporarily hide symptoms, trigger longer recovery, or lose ring-buffer evidence.

Chaining the 10 Commands into a Complete Investigation

Assume monitoring reports API timeout with simultaneous node load increase; follow this loop: uptime — confirm recent load rise, record host uptime. top — observe CPU, memory, task states; determine if whole-machine CPU busy or many D -state tasks. vmstat — use r, b, wa, si/so to differentiate CPU queuing, I/O wait, and paging. mpstat -P ALL — judge single-core vs multi-core, user vs kernel mode. pidstat — locate persistently anomalous processes/threads, not one-shot rankings. iostat — verify block device queue and latency align with fault window. ss — confirm listeners, connection states, socket queues; exclude request-not-arriving or connection pile-up. df -hT, df -i — check capacity, inodes, real mount points. journalctl — align app exits, retries, dependency timeouts, deploy info. dmesg or journalctl -k — check OOM, device, filesystem, kernel anomalies.

Each step should produce a "what to check next" judgment. Example: vmstat shows high b and wa; iostat simultaneously shows target device await and queue rising; kernel logs also show device timeouts → storage path evidence converges. If only iostat %util is high while app latency, queues, and logs are normal, evidence is insufficient to conclude.

Three Common Misjudgments

Misjudgment 1: High Load = Insufficient CPU

Load includes uninterruptible-sleep tasks. Must combine mpstat CPU time and

vmstat

r

/ b. If CPU mostly idle but b high, adding CPU usually won't help.

Misjudgment 2: Low Free Memory = Memory Leak

Linux uses memory for cache. Focus on MemAvailable, process RSS/anonymous memory, cgroup limits, PSI, swap activity, and time trends. Post-OOM-kill free memory cannot retroactively reflect pre-fault state.

Misjudgment 3: Port Unreachable = Firewall Issue

First use ss -lntp to confirm service listening and correct bind address; then check routing, load balancer, security groups, firewall, and return path. Blindly opening ports may expand attack surface and won't fix unlistening service or app blocking.

Evidence Conditions Required Before Executing Fixes

Commands help locate; they don't grant immediate fix authority. For actions involving data deletion, service restart, log cleanup, kernel parameter changes, firewall, database params, Docker/K8s resources, cluster scaling, or failover, at minimum satisfy:

Impact Scope : clarify hosts, instances, connections, data, downstream, and redundancy; assess trigger of leader election, rescheduling, or traffic shift.

Pre-execution Checks : confirm healthy replicas, remaining capacity, in-flight tasks, error rates, latency, dependency status, and operation permissions.

Backup Method : save original configs, versioned manifests, key logs, diagnostic data; for stateful services follow product-specific official consistency backup procedures.

Canary or Dry-run : validate first in test env, single instance, or few nodes; dry-run checks change objects but cannot replace real business validation.

Verification Method : simultaneously verify resource metrics, service health, error rates, throughput, P95/P99 latency, and key business functions—not just command exit codes.

Rollback Plan : prepare original configs, artifacts, and cutover path; define metric thresholds and responsible person to trigger rollback.

Restarts need extra caution. They may release resources, clear connections, and temporarily restore service, but also lose /proc scene, memory objects, and partial ring logs. If restart is mandatory: drain traffic first, preserve evidence, confirm healthy replicas and capacity, and process only one instance gradually. kill -9 prevents graceful cleanup; avoid as routine.

Post-Fix Verification and Post-Mortem

After fix, re-run the same sampling set using identical time windows and statistical calibers. At minimum verify:

Load, run queue, CPU types, paging, I/O wait returned to historical baselines.

Target process CPU, memory, I/O, context switches stable.

Disk await, queue, capacity, inodes, kernel errors recovered.

Listeners and connection states normal; traffic genuinely restored, not just dropped due to drain.

Error rates, throughput, P95/P99 latency, business functions meet SLO.

Fix didn't shift pressure to other nodes, disks, or downstream services.

Post-mortem must record original commands, sampling times, key outputs, change timeline, direct cause, root cause, fix, and rollback result. Avoid entries like "CPU high, restart fixed" or "disk full, cleanup done." Such records cannot help next on-call engineer judge similar issues nor prove operational safety.

The value of these 10 commands lies not in memorizing every parameter, but in establishing a fixed sequence: first confirm time and overall state, then differentiate resource types, then locate objects, finally close loop with service and kernel logs. Troubleshooting efficiency comes from evidence progressively converging, not from running more commands. As long as each step answers "what did I see, what does it prove, what does it not yet prove," most common production issues won't stay in the guessing stage.