Mastering Linux Performance: Metrics, Tools, and Optimization Techniques

Key Performance Metrics

CPU usage (user, system, iowait, soft/hard interrupts, steal)

Average load (runnable processes, not a direct CPU usage percentage)

Process context switches (voluntary vs. involuntary)

CPU cache hit rate

Memory usage (VIRT, RES, SHR, %MEM), buffer/cache, swap, OOM handling

Typical Analysis Workflow

Identify the bottleneck type: CPU‑bound, I/O‑bound, memory‑bound, high load, zombie processes, etc.

Get a quick overview with uptime or vmstat to see load average and context‑switch rates.

Drill down to the offending process with pidstat (‑u for CPU, ‑d for I/O, ‑r for memory, ‑w for context switches).

If top / pidstat cannot locate the culprit, use perf (e.g., perf top, perf record, perf report -g) or strace to trace system calls.

For I/O‑related problems, inspect /proc/interrupts, dstat, and iostat to correlate interrupt spikes with disk activity.

For memory pressure, combine free, top, vmstat and BCC tools such as memleak, cachestat, cachetop.

CPU‑Specific Guidance

CPU context switches save the current task’s registers and load the next task’s registers. System calls cause two switches (user→kernel and kernel→user). Excessive switches add overhead. Reduce them by:

Using threads instead of processes when possible.

Binding processes or threads to specific CPUs (CPU affinity).

Adjusting niceness or using cgroups to limit CPU share.

vmstat 5          # output every 5 seconds

pidstat -w 5      # per‑process voluntary/involuntary switches

When high CPU usage appears without a clear hot process, check the run‑queue length (column r) and interrupt count ( in). A large run‑queue indicates many short‑lived tasks or frequent context switches.

Memory Management

Linux separates kernel and user virtual address spaces. Allocation uses brk() for small blocks (<128 KB) and mmap() for larger regions. Minor page faults occur on first access; major faults trigger disk I/O (swap or file read).

Memory reclamation includes LRU cache eviction, swapping anonymous pages, and OOM killing. Control swap aggressiveness with /proc/sys/vm/swappiness and inspect zone thresholds ( /proc/zoneinfo pages_min{{{ }}, pages_low, pages_high) to understand pressure.