How to Diagnose a Suddenly Lagging Linux Server: Step‑by‑Step Ops Checklist

1. Clarify the Symptom

Record the exact time, duration, scope (whole machine, single service, interface, or client batch), SSH responsiveness, latency distribution, recent deployments, scaling events, batch jobs, backups, or security scans, and the source of the alert (application, system, load balancer, or user).

2. Collect a Read‑Only Snapshot

Run low‑impact, read‑only commands and store their output in a timestamped directory (e.g., DIAG_DIR="/var/tmp/diag-$(hostname)-$(date +%Y%m%d-%H%M%S)"), then:

mkdir -p "$DIAG_DIR"
date > "$DIAG_DIR/date.txt"
hostnamectl > "$DIAG_DIR/hostnamectl.txt"
uptime | tee "$DIAG_DIR/uptime.txt"
free -h | tee "$DIAG_DIR/free.txt"
vmstat 1 10 | tee "$DIAG_DIR/vmstat.txt"

Set restrictive permissions ( chmod 700 "$DIAG_DIR") and avoid writing to a full partition.

3. Initial Assessment – CPU, Memory, I/O, Network

Start with vmstat 1 10 and focus on the fields below. Compare values to the number of logical CPUs ( nproc or lscpu). r – runnable processes; high values relative to CPUs indicate CPU queuing. b – blocked I/O; high values suggest I/O wait. si/so – swap in/out; non‑zero values mean active swapping. us/sy – user and system CPU usage. wa – CPU idle time spent waiting for I/O (not a direct disk utilization metric). st – stolen time (relevant for VMs).

CPU Deep‑Dive

If r is persistently high, run: mpstat -P ALL 1 5 Interpret %usr (compute‑bound), %sys (system calls, locks, network), and %soft (soft interrupts). Identify the hottest core and trace the responsible process:

pidstat -u 1 10
ps -eo pid,ppid,user,stat,psr,pcpu,pmem,etime,comm,args --sort=-pcpu | head -n 30

For detailed profiling, use perf top or perf record on the target PID, limiting sampling time (e.g., timeout 30 perf top -p 18472).

Memory Deep‑Dive

Check overall memory and swap:

free -h
grep -E 'MemAvailable|SwapTotal|SwapFree|Dirty|Writeback|Slab|SReclaimable' /proc/meminfo
vmstat 1 10

Find memory‑hungry processes:

ps -eo pid,user,rss,vsz,pmem,etime,comm,args --sort=-rss | head -n 30
pidstat -r 1 10

For containers, inspect cgroup v2 limits:

cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.events

Never use echo 3 > /proc/sys/vm/drop_caches as a regular fix.

Disk and Filesystem

Verify space and inode usage:

df -hT
df -ih
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS

Monitor device latency and queue length: iostat -xz 1 10 Identify top I/O processes:

pidstat -d 1 10
iotop -oPa

Check kernel logs for I/O errors:

journalctl -k | grep -Ei 'I/O error|timeout|reset|nvme|ext4|xfs|nfs'

Network

Collect interface and TCP statistics:

ip -s link
sar -n DEV 1 10
sar -n TCP,ETCP 1 10
ss -s

Identify hot remote endpoints:

ss -tanp
ss -lntp
ss -tan state established | awk '{print $5}' | sort | uniq -c | sort -nr | head

Measure DNS and TLS latency for a specific endpoint:

dig +time=2 example.com
curl --write-out 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}
' -o /dev/null -s https://service.example.com/health

4. Application‑Level Checks

Inspect service status and recent logs:

systemctl status example.service
journalctl -u example.service --since "2026-07-13 14:00:00" --until "2026-07-13 15:00:00"

Look for thread‑pool saturation, connection‑pool wait times, downstream latency spikes, GC pauses, or lock contention. Verify that health‑check endpoints reflect true readiness.

5. Resource Limits

Check open‑file limits and current descriptor count for the main process:

SERVICE_PID=$(systemctl show -p MainPID --value example.service)
cat /proc/$SERVICE_PID/limits | grep 'Max open files'
lsof +L1 | wc -l

Confirm ulimit -n and kernel parameters for TCP ports ( sysctl net.ipv4.ip_local_port_range) and connection‑tracking ( /proc/sys/net/netfilter/nf_conntrack_*).

6. Container / VM Context

For cgroup v2, view CPU quota and throttling:

cat /sys/fs/cgroup/cpu.max
cat /sys/fs/cgroup/cpu.stat

For VMs, monitor steal time with mpstat and compare to host‑level metrics.

7. Historical Data Recovery

If sysstat is enabled, retrieve past metrics for the fault window (adjust paths and timestamps as needed):

sar -u -f /var/log/sa/sa13 -s 14:20:00 -e 14:35:00
sar -q -f /var/log/sa/sa13 -s 14:20:00 -e 14:35:00
sar -r -S -f /var/log/sa/sa13 -s 14:20:00 -e 14:35:00
sar -d -p -f /var/log/sa/sa13 -s 14:20:00 -e 14:35:00
sar -n DEV,TCP,ETCP -f /var/log/sa/sa13 -s 14:20:00 -e 14:35:00

8. Evidence‑Based Root‑Cause Confirmation

Build a timeline that links observed spikes (e.g., high w_await, increased r queue, specific process I/O) with application logs and downstream failures. Only conclude a root cause when time, object, and mechanism align.

9. Mitigation and Repair

Apply low‑risk stop‑gap actions first (pause non‑critical batch jobs, reduce concurrency, adjust thread‑pool limits). If a service restart is required, isolate one instance, verify health checks, then run: systemctl restart example.service Ensure backups, configuration snapshots, and rollback plans are in place before any high‑risk changes (kernel parameters, cgroup limits).

10. Post‑Repair Validation

Confirm that system‑level metrics have returned to baseline, process‑level resources have stabilized, and application‑level SLAs (throughput, error rate, P50/P95/P99 latency) are met. A simple end‑to‑end health probe:

curl --fail --silent --max-time 5 https://service.example.com/health

11. Post‑Mortem and Monitoring Gaps

Document the incident timeline, impact, evidence chain, mitigation steps, and any missing observability (e.g., lack of I/O latency alerts, insufficient connection‑track metrics). Add or refine Prometheus alerts, capacity‑planning dashboards, and runbooks to prevent recurrence.

12. Consolidated Troubleshooting Checklist

Define fault window and scope.

Save current monitoring snapshots.

Run vmstat to locate CPU, I/O, or swap pressure.

Drill down with mpstat, pidstat, and ps.

Inspect memory via free, /proc/meminfo, PSI, and OOM logs.

Check disk space, inode usage, iostat, and device errors.

Analyze network stats with ip -s link, sar -n DEV, ss, and DNS/TLS latency tests.

Review application logs, thread/connection pools, and downstream dependencies.

Validate file‑descriptor and process limits.

Consider container/VM cgroup quotas and host steal time.

Correlate with historical sar data if available.

Build an evidence‑based root‑cause chain.

Apply mitigation, then perform a controlled restart if needed.

Verify recovery across system, process, application, and user layers.

Document the incident and close monitoring gaps.