Skip to content
Four Swords of Linux Performance Analysis: top, vmstat, iostat, sar in Practice
Operations 75 min read

Four Swords of Linux Performance Analysis: top, vmstat, iostat, sar in Practice

This article reveals how top, vmstat, iostat, and sar all derive metrics from shared kernel counters, explains their common pitfalls like first-sample averages, maps each tool's fields to /proc sources, demonstrates baseline configuration and high-frequency commands, and walks through three real-world cases where contradictory tool outputs led to root-cause discovery.

MaGe Linux Operations
MaGe Linux Operations
MaGe Linux Operations

Core Thesis: Four Tools, One Kernel Counter Source

The article opens by correcting a widespread mental model: top, vmstat, iostat, and sar are not four independent instruments measuring CPU, memory, and disk separately. Instead, the Linux kernel continuously accumulates events into a set of counters exposed as plain text via /proc and /sys. All four tools do the same thing — open files, read numbers, subtract the previous snapshot, and divide by the sampling interval. Their only differences are which counters they read, the default sampling interval, and whether they can access historical data.

Part 1: Foundations — CPU Accounting and Common Pitfalls

1.1 Kernel CPU Time: Jiffies and /proc/stat

The first line of /proc/stat is the root of the entire analysis system. Each field is a cumulative count of jiffies since boot (1 jiffy = 1/ CONFIG_HZ seconds; x86 commonly uses HZ=250 or 1000). The fields are

user nice system idle iowait irq softirq steal guest guest_nice

. The article demonstrates a manual verification: reading total jiffies twice with a one-second gap, computing the difference, and confirming it matches cores × HZ (e.g., 14 cores × 100 = 1400 tick/s). This establishes the denominator for any CPU percentage: cores × HZ × seconds.

Three engineering conclusions follow:

%CPU in top can exceed 100% because top defaults to Irix mode (percentage of a single core). An 8-thread Java process at full load shows 800%. Press Shift+I to switch to Solaris mode (divided by core count).

iowait means "CPU idle and at least one task waiting for IO", not "disk busy". The kernel increments iowait only when a CPU has no runnable tasks, enters idle, and the run queue has a task with the in_iowait flag set. If other runnable tasks exist, iowait stays zero even if the disk is 100% utilized.

steal time is not your fault. It represents hypervisor giving physical CPU time to another vCPU. Steal > 5% indicates severe host overcommit; application-level tuning is useless — migrate or increase quota.

1.2 The First-Line Trap

All /proc -based delta tools lack a previous value for the very first sample. vmstat and iostat use the boot-time cumulative average; sar skips the first output entirely. The article shows vmstat 1 3 where the first line reports us=3 sy=2 id=94 (40-day average) while the second line shows the real us=18 sy=6. iostat -x 1 2 similarly shows a first line with w/s=18.40 (lifetime average) versus w/s=3180 (current). Any script that runs a single iteration and does not discard the first line will underestimate metrics by orders of magnitude — a common reason automated alerts never fire. top -b -n 1 suffers the same issue for its CPU summary line while memory and process list are real-time, making the bug harder to spot.

1.3 Production Baseline: No Baseline, No Anomaly

Performance analysis most often skips recording the normal state. The same wa=8% is critical on a memory-only gateway but normal during a database backup window. The article provides a concrete baseline environment (Rocky Linux 9.4, dual Xeon Gold 6330 112 logical cores, 512 GiB RAM, NVMe RAID10, Kafka+Flink mixed workload) and shows how to use sar history to retrieve week-over-week baselines with a bash loop extracting daily 10:00–11:00 CPU averages.

Part 2: Field Alignment — Mapping Every Column to Its Kernel Counter

2.1 vmstat: r, b, si, so, in, cs

r

= run queue length (

/proc/stat
procs_running

minus 1 for vmstat itself). Sustained r > logical cores indicates true CPU saturation. b = tasks in uninterruptible sleep (D state) from procs_blocked. Persistent b > 0 means tasks are stuck on IO; high wa is a symptom, not the cause.

Finding D-state processes: ps -eo pid,stat,wchan:24,comm | awk '$2 ~ /D/'. The WCHAN column reveals the exact kernel function (e.g., submit_bio_wait for block layer, rpc_wait_bit_killable for NFS). si / so = swap in/out rate (KB/s). Sustained non-zero si on a production server is a failure — memory pressure causing thrash. in = interrupts/sec, cs = context switches/sec. cs ≈ in (tens of thousands) is normal; cs an order of magnitude higher with high sy suggests lock contention or excessive threads.

2.2 iostat -x: %util Is the Most Misread Column

%util

= (device busy time) / (total time), derived from /sys/block/<dev>/inflight non-zero samples. For single-queue SATA/SAS disks it approximates "disk busy". For NVMe multi-queue devices, %util=100 only means "at least one IO in flight at every instant"; queue depth can be 1024, so throughput may be only 20% of capacity. To judge NVMe saturation, watch aqu-sz (average queue depth) and r_await / w_await: aqu-sz approaching device nr_requests and await rising significantly (>5ms) indicates a true bottleneck. The article shows an example where aqu-sz=41.2, w_await=1.84ms, %util=99.8 — queue has headroom, the device is just continuously busy. await = queue wait + service time; svctm is unreliable at high utilization and deprecated since kernel 4.18; use aqu-sz instead.

2.3 top Memory: SHR, RES, VIRT and Memory Accounting

VIRT

= virtual address space (includes mmapped model files, reserved heap). For Java/inference processes it can be tens of GB — does not represent actual memory usage . RES = resident physical memory. SHR = potentially shared portion (shared libraries, tmpfs mappings).

Summing RES across multiple processes double-counts shared pages. A parent forking four workers each showing RES=2g may only consume 2.2g physically. True memory pressure must be read from

/proc/meminfo
MemAvailable

(also shown as available in free -m). Using MemFree is the most common novice mistake — PageCache legitimately consumes free memory, so a few GB free is healthy.

2.4 sar: The Only Tool with Historical Reach via sadc

sar

reads binary history files ( /var/log/sa/saDD) written periodically by sadc (driven by sysstat-collect.timer or cron). Default interval is 10 minutes ( INTERVAL in /etc/sysconfig/sysstat), which is too coarse for 15-minute P99 spikes — necessitating a separate high-frequency collector. The article shows sar -u -r -b -q -f /var/log/sa/sa21 output and notes that sar 's %iowait, tps etc. come from the same /proc/stat and /proc/diskstats counters as vmstat and iostat; cross-validation requires matching sampling windows.

Part 3: Baseline Collection — Making the Four Swords See the Same Timeframe

3.1 Dual-Layer Configuration: sysstat Collection Layer + Kernel Counter Layer

Application/collection layer ( /etc/sysstat/sysstat ): HISTORY=30 (retain 30 days for year-over-year), COMPRESSAFTER=6, INTERVAL=60 (1-minute granularity),

SADC_OPTIONS="-S DISK -S INT -S XDISK -S POWER -S IPV6 -S XFS"

(enable detailed subsets). Reload with

systemctl daemon-reload && systemctl restart sysstat-collect.timer

. Verify actual interval via

sadf -d /var/log/sa/sa21 -- -u | awk -F';' 'NR>3{print $3}' | head -5

showing 60-second steps.

Kernel counter layer ( /etc/sysctl.d/90-perf-observability.conf ): kernel.kptr_restrict=0, kernel.perf_event_paranoid=1 (allow CPU events and tracepoints but block raw tracepoint), kernel.sched_schedstats=1 (required for pidstat -w voluntary switches), vm.nr_hugepages=0, kernel.numa_balancing=1, TCP retry defaults. perf_event_paranoid=1 is the production sweet spot; 2 (common default) disables kernel tracepoints, 3 disables almost everything.

Disk scheduler check: NVMe should be none, SATA SSD mq-deadline. The article finds sda using bfq (desktop scheduler) which adds latency and distorts await — a setup for Case 2.

3.2 Four-Dimensional Breakdown of the Collection Config

Must-change items: INTERVAL=60 (else 10-min granularity misses short spikes), HISTORY=30 (else no week-over-week), perf_event_paranoid=1 (else perf permission denied), sched_schedstats=1 (else pidstat -w loses voluntary switches).

Coupled items (changing one forces another): Lowering INTERVAL from 600 to 60 increases disk usage 10x (single sa file ~4MB → ~40MB; 30 days ≈ 1.2GB raw, ~250MB compressed). Collection overhead: sadc single run ~94ms on 112-core/1800-process machine (0.16% CPU), but -S INT reads /proc/interrupts (~20ms on 128-queue NIC). Monitoring alignment: Prometheus node_exporter scrapes every 15s; to compare with sar 's 60s points, use max_over_time(...[5m]) not avg.

Tuning formulas (no magic numbers): INTERVAL ≤ shortest_anomaly_duration / 2 (Nyquist). For 15-min period spikes, 60s gives 15 points — barely enough for trend, not shape. If spike lasts 20-30s, need INTERVAL ≤ 10; deploy a separate high-frequency sadc instance rather than lowering global interval (6x file growth). Disk budget: daily_increment = sample_size × (86400/INTERVAL); retention = daily_increment × days × 0.2. Example: 640KB/sample at 10s interval, 3 days → ~3.2GB.

Absolute red lines: Never disable kernel.sched_schedstats (saves <0.3% but breaks pidstat -w and perf sched). Never set perf_event_paranoid=-1 (allows arbitrary kernel address sampling — privilege escalation surface). Never change kernel.pid_max without reboot (PID wrap-around conflicts). Never delete saDD while keeping sarDD ( sar -f reads saDD; sarDD is only a text summary).

Part 4: High-Frequency One-Liners — Atomic Operations for Troubleshooting

4.1 Locate CPU Consumers: From Summary to Thread

top -b -n 2 -d 0.5 -H -o %CPU | awk '/^top -/{n++} n==2' | head -15

— -H shows threads; Java/Go %CPU is sum of all threads, without -H you only see a single huge percentage. Convert high-CPU TID to hex ( printf '%x') and search jstack for nid.

Manual 3-second delta from /proc/<pid>/stat fields 14+15 ( utime+stime), compute per-core percentage using CLK_TCK (from getconf CLK_TCK). More accurate than top for short-lived processes (which can show 12000% in top). pidstat -p 12051 -u -w -I 1 3 — shows per-thread %usr/%system/%CPU, CPU affinity, voluntary ( cswch/s) and non-voluntary ( nvcswch/s) context switches. nvcswch/s > 100 with low cswch/s indicates CPU contention, not IO wait.

4.2 Locate IO Consumers: From Disk to Process to File

iotop -o -b -n 2 -d 1 | tail -20

— IO> column = fraction of time process spent waiting for block layer. SWAPIN > 0 is direct evidence of memory pressure.

Without iotop: read

/proc/<pid>/io
read_bytes

/ write_bytes (bytes actually submitted to block layer, excluding PageCache hits) over a 2-second interval. rchar / wchar are syscall bytes including cache hits — using them overstates IO by orders of magnitude.

Compare wchar vs write_bytes: ratio ≈1 means almost no cache hits (typical of O_DIRECT or frequent fsync); ratio ≈0.01 means writes go to PageCache (IO pressure appears later via

vmstat
bo

column).

NVMe queue depth: cat /sys/block/nvme0n1/inflight (read/write in-flight), nr_requests (queue limit), scheduler (should be none). In-flight approaching limit = true saturation.

4.3 Locate Memory Consumers: Available, PageCache, Slab

free -m

and

awk '/^(MemTotal|MemFree|MemAvailable|Dirty|Writeback|Slab)/ {printf "%-14s %7.2f GB
", $1, $2/1048576}' /proc/meminfo

. MemAvailable is the real "allocatable" memory (already excludes unreclaimable slab and reserves). SUnreclaim > 5% of total RAM warrants kernel leak investigation. Dirty small means writeback is timely. slabtop -o -s c | head -10 — if a cache shows ACTIVE/USE near 100% and CACHE SIZE growing continuously, suspect leak; use kmemleak or slabinfo -X.

4.4 Monitoring Integration: PromQL Matching Tool Semantics

Alert rules must use the same denominators as the CLI tools. Examples: RunQueueHigh:

node_load1 / count without(cpu,mode) (node_cpu_seconds_total{mode="idle"}) > 2

for 10m. BlockedTasks: node_procs_blocked > 5 for 5m (directly maps to vmstat b). DiskUtilHigh:

rate(node_disk_io_time_seconds_total[5m]) > 0.95 and rate(node_disk_io_time_weighted_seconds_total[5m]) / rate(node_disk_io_time_seconds_total[5m]) > 2

— the second term is aqu-sz; using only io_time (i.e., %util) misfires on NVMe. MemAvailableLow:

node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1

for 5m. IowaitWithBlocked:

rate(node_cpu_seconds_total{mode="iowait"}[5m]) > 0.05 and on(instance) node_procs_blocked > 0

— avoids false alerts when iowait is high but no tasks are actually blocked.

4.5 PSI: Kernel 4.20+ Pressure Stall Information

PSI measures "time tasks were stalled waiting for a resource" — the dimension the four tools miss.

cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory

shows some (at least one task stalled) and full (all runnable tasks stalled) averages over 10s/60s/300s. In the article's example, io full avg10=31.42 means 31% of the last 10 seconds all runnable tasks were waiting for IO — far more severe than wa=6%. cpu full staying zero is normal (CPU pressure always leaves some task runnable). A

watch -n2 'paste /proc/pressure/io <(iostat -x nvme0n1 1 2 | tail -1)'

juxtaposes PSI and iostat: when %util=99.8 but io some avg10=2.10, the device is busy but tasks aren't stalled — classic %util false positive.

Part 5: Three Deep-Water Fault Loops — Resolving Contradictions

5.1 Case 1: 15-Minute P99 Spike Invisible to 10-Minute sar

Symptom: Flink write P99 jumps from 18ms to 48ms every ~15 minutes for 40-60s. Grafana shows flat CPU/memory/network; manual iostat only shows slight w/s rise.

Confirm periodicity: Grep Flink checkpoint logs → exactly 17:00, 17:15, 17:30... points to a cron-like trigger (Flink checkpoint interval, Kafka log.roll.ms, system cron).

Use sar to exclude system cron: sar -u -b -q -f /var/log/sa/sa21 -s 16:44:00 -e 16:47:00 shows everything normal ( %iowait=0.41, blocked=0). 10-minute granularity smears the 60-second spike into invisibility. Relying solely on sar would misdirect investigation to the application.

Deploy high-frequency collector:

sadc -S CPU -S DISK -S XDISK -S MEM -S QUEUE 5 720 /dev/shm/hisa/sa_hf

(5s interval, 1 hour, writes to tmpfs to avoid self-interference). Next spike captured with sar -b -q -f /dev/shm/hisa/sa_hf -s 17:14:30 -e 17:16:30 reveals wtps collapsing from 31880 to 4820 and dtps (discard/flush requests) surging from 100 to 38420 — block layer submissions stalled.

Track dirty pages and writeback: Sampling

/proc/vmstat
nr_dirty

and nr_writeback every 3s shows nr_dirty skyrocketing from 412k pages (1.6GB) to 21.8M pages (83GB) in 12 seconds. nr_dirty_threshold (from vm.dirty_ratio=8 on 512GB RAM = 41GB) is the hard limit. Background writeback kicks in, but sequential flush competes with Kafka append writes for the block queue, queuing fsync requests (Kafka segment roll, Flink checkpoint) and inflating P99.

Root cause: dirty_ratio=8 allows 41GB dirty accumulation. Kafka log.roll.ms=900000 (15min) triggers segment roll + fsync; Flink checkpoint also 15min — dual resonance.

Fix (low to high risk): (1) Switch to absolute byte limits: vm.dirty_background_bytes=1073741824, vm.dirty_bytes=4294967296, vm.dirty_expire_centisecs=1000, vm.dirty_writeback_centisecs=100, zero out ratio counterparts. (2) Stagger Kafka: log.roll.ms=1800000 (30min), log.flush.interval.messages=100000. (3) Verify: post-fix sar shows stable wtps, dtps peak 108, nr_dirty peak 280k pages (1.1GB). P99 48ms → 19ms, P99.9 210ms → 41ms. Rollback: delete sysctl file, restore ratios.

5.2 Case 2: %util=100, await=200ms, NVMe Swap Fails

Symptom: Log search P99 40ms → 900ms. iostat -x shows

sda
%util=100

, w_await 1.2ms → 210ms. Ops replace SATA SSD with NVMe; await drops to 180ms, problem persists.

Don't trust %util; check queue depth and request size: iostat -x -m 1 3 sda nvme2n1 shows NVMe w_await=0.92ms (device no longer bottleneck) but %util=100 still. aqu-sz=0.38, wareq-sz=0.3MB (300KB/request). Queue depth <1 means no queuing at all. 1.2MB/s write bandwidth driving NVMe to 100% %util is physically impossible unless time is spent in request processing path, not data transfer.

Block layer stats confirm: /sys/block/nvme2n1/stat last field time_in_queue=842 vs io_ticks=98041 — queue time far less than busy time; %util = io_ticks / interval counts any in-flight request as 100% busy.

Real source is CPU/scheduler: vmstat shows r=42, us=79, id=2, cs=41208/s. PSI: io some avg10=1.20 (no IO pressure), cpu some avg10=88.42, cpu full avg10=42.18. pidstat -w reveals Java processes with cswch/s=18420 but only 16% CPU — voluntary switches indicate lock contention or sync IO. PSI shows no IO pressure, so it's locks.

perf confirms: perf record -F 999 -g -p 12043 -- sleep 30 → __lll_lock_wait (38.4%) + ObjectMonitor::enter (22.2%) = 60% lock contention. Root cause: log service uses a single synchronized covering serialization + disk write; write threads scaled from 16 to 64, lock contention exploded. SATA await=210ms was a symptom — lock holder doing sync write, 63 waiters piled up, block queue grew. NVMe shortened hold time (210ms → 0.92ms) but lock remained.

Fix: Application: striped locks (64 segments), async flush via single-thread executor + group commit. System: udev rule to set sda scheduler to mq-deadline (runtime echo fails due to module conflict),

fstab
noatime,nodiratime,data=ordered

(explicit ordered prevents unsafe writeback). Result: P99 900ms → 42ms, r 42 → 4, cs 41208 → 4820, __lll_lock_wait 38.4% → 1.2%. Rollback: app version rollback, udev rule removal, mount -o remount.

5.3 Case 3: load=180, CPU 85% Idle, Disk Quiet

Symptom: 32-core NFS client, load average 150-200,

top
id=85%

,

iostat
%util<20%

, network <10%. Occasional stalls, no errors.

Deconstruct load average: Linux load = procs_running + procs_blocked (exponential moving averages over 1/5/15min, 5s sampling). It includes D-state tasks — so high load with idle CPU is mathematically inevitable, not anomalous. awk '/^procs_running|^procs_blocked/{print}' /proc/stat → procs_running=6, procs_blocked=174. /proc/loadavg last field 6/2104 = running/total tasks.

Identify D-state wait channel:

ps -eo pid,stat,wchan:28,etime,comm --sort=-etime | awk '$2 ~ /^D/'

→ all stuck in rpc_wait_bit_killable (NFS RPC reply wait). WCHAN alone pinpoints subsystem.

NFS client stats: nfsstat -c → retrans=482104 / calls=18421044 = 2.6% retransmit rate. Threshold: >1% degrades latency, >5% indicates clear network/server issue. nfsstat -m shows timeo=600 (60s timeout), retrans=2 → worst-case 180s per read. Explains 4-hour stuck tar.

Network vs server: ss -tin dst 10.20.30.41 → retrans:14/4210 (0.33%), rtt:0.184ms — network healthy.

/proc/self/mountstats
READ

line: queue=2184μs, exe=1204880μs — queue 2ms, execution 1.2s. Bottleneck is server storage, not RPC queue.

Server confirmation: On 10.20.30.41: iostat -x -m 1 3 → sdb/sdc aqu-sz=182, r_await=48ms, w_await=92ms, %util=100. 8×7200rpm SATA RAID6 theoretical IOPS ~960, actual r/s+w/s≈16000 — 16x overload. Client concurrency: lsof +D /data/nfs → 842 tar processes. NFSv4.1 session slots default 16, so RPCs queue, timeout, retransmit, processes enter D-state — positive feedback loop.

Fix (triage to root): (1) Immediate: systemctl set-property --runtime user.slice IOWeight=50; kill excess tar processes. (2) Mount options: mount -o remount,timeo=60,retrans=1 /data/nfs (keep hard — soft causes silent data corruption). (3) Server: export with sync (never async), plan storage upgrade. Result: load 184 → 12, procs_blocked 174 → 0, NFS retrans 2.6% → 0.02%, server r_await 48ms → 4.2ms, tar batch 4h+ → 38min. Rollback: remount original options, exportfs -ra, cgroup runtime property auto-expires.

5.4 Common Conclusion: Contradiction Is the Answer

All three cases entered via different misleading aggregates ( wa, %util, load) but shared the same error path: taking an aggregated metric at face value without asking "what exactly does this counter measure?" The article summarizes a lookup table: Case 1 — sar normal because 10-min granularity smeared spike → deploy 5s collector, watch nr_dirty; Case 2 — %util=100 means device had in-flight IO, not saturation → check aqu-sz and wareq-sz; Case 3 — load=184 includes 174 D-state tasks → check procs_blocked and WCHAN. The correct workflow: form a hypothesis, then pick the tool that can falsify it. vmstat b falsifies CPU hypothesis; iostat aqu-sz falsifies disk saturation; ps WCHAN falsifies "kernel busy"; sar falsifies "always been this way".

Part 6: One-Script Black Box — 10-Second Full Scene Capture

perf-snapshot.sh

runs vmstat, mpstat, iostat, pidstat at 1s intervals for 10s (background), plus instantaneous snapshots of load, procs, PSI, D-state tasks with WCHAN, memory, dirty pages, slab top10, device schedulers. Output directory contains vmstat.txt, mpstat.txt, iostat.txt, pidstat.txt, snapshot.txt, blocked.txt, kernel.txt, and a summary.txt with average r / b and PSI some avg10 lines. Safety features: set -euo pipefail, quoted variables, --dry-run mode. Usage: ./perf-snapshot.sh /var/tmp/perf-case1 10.

Part 7: Production Change Risk Matrix

Changes ordered by risk: INTERVAL=60s: +40MB/day disk, no outage, rollback via config + timer restart — Low. perf_event_paranoid=1: widens attack surface slightly, no outage, instant sysctl -w rollback — Low. dirty_bytes limits: global writeback behavior, no outage, instant sysctl -w rollback — Medium.

IO scheduler switch (udev): device latency characteristics, no outage, delete udev rule + trigger — Medium. mount -o remount: mount-point IO brief hang (80-300ms), remount back — Medium-High. log.roll.ms: segment count/recovery time, requires broker restart, revert + restart — Medium-High.

NFS timeo/retrans: all NFS apps brief hang, remount back — High.

High-risk operation checklist (exemplified by mount -o remount): define blast radius, pre-checks ( findmnt, lsof, nfsstat -m), backup ( cp /etc/fstab, record findmnt -no OPTIONS), execute in low-traffic window avoiding batch job start times, canary on one non-critical node for 24h, one-command rollback with same hang window.

7.1 Three Inviolable Red Lines

dirty_bytes cannot be smaller than max single write size. If app write() exceeds dirty_bytes, it falls back to synchronous write (microseconds → milliseconds). Formula:

dirty_bytes ≥ max_write_size × concurrent_write_threads × 2

.

vm.dirty_ratio and vm.dirty_bytes are mutually exclusive. Both set in /etc/sysctl.d/ — lexical file order decides winner, not content. Behavior becomes dependent on filename, a stealth failure source.

%util must never be the sole alert condition. Proven in Case 2: NVMe %util=100 with aqu-sz=0.38 is perfectly healthy. Alerts must combine io_time and io_time_weighted (i.e., aqu-sz) or use PSI directly.

Part 8: Seven Core Engineering Laws

All sampling tools' first frame is the boot-time average — discard it. vmstat 1 1, iostat 1 1, top -b -n 1 are the three most common traps. Compare first line to subsequent lines; order-of-magnitude gap confirms boot average.

Know your denominators: %CPU per core, %util numerator is non-zero in-flight, load includes D-state. These three denominator errors cover 80% of production misdiagnoses. Before using any metric, ask "what is its denominator?"

High iowait ≠ slow disk; low iowait ≠ fast disk. iowait only increments when CPU is idle AND a task waits for IO. On multi-core, as long as one core runs other tasks, iowait can be fully masked. Judge disk with iostat await/aqu-sz or PSI io full.

MemFree is not available memory; sum of RES is not actual usage. Trust only MemAvailable; for per-process usage use PSS ( /proc/<pid>/smaps_rollup), not RES.

No baseline, no threshold. Same numbers mean opposite things on different role machines. Keep at least 7 days of sar history and align monitoring scrape intervals with CLI sampling windows, else "Grafana says 95%, sar says 62%" debates never converge.

When tools contradict, suspect the metric definition first, then the data, last the tool. All three cases proved contradiction is the strongest clue — it precisely identifies "you are interpreting a correct number in the wrong dimension".

Write rollback before you write execution. perf-snapshot.sh is read-only (zero risk); any remount, sysctl -w, scheduler change must have the rollback command in the same block, not recalled later.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

troubleshootingsystem monitoringtopsariostatvmstatLinux performancekernel counters
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.