Comprehensive Guide to Linux Performance Optimization

Performance metrics

Throughput and latency are the two core indicators for high‑concurrency, low‑latency services. A performance problem appears when system resources reach a bottleneck while request handling remains too slow.

Average load

Average load is the average number of processes in runnable or uninterruptible (I/O‑waiting) state over a time interval. It is unrelated to CPU utilization percentages. Uninterruptible processes are those blocked in kernel‑mode I/O. Monitor with uptime and set a threshold such as 70 % of the CPU count.

CPU context switching

CPU context switching saves the current task’s registers and program counter, then loads the next task’s context. Three categories are described:

Process context switch

Thread context switch

Interrupt context switch

A system call performs two switches: user → kernel (save user registers) and kernel → user (restore them). The call is a privilege‑mode switch, not a full process switch.

Observe system‑wide switches with vmstat 5:

vmstat 5          # output every 5 seconds
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 103388 145412 511056    0    0    18    60   1   1  2  1 96  0  0

Key columns: cs (context switches per second), in (interrupts per second), r (runnable queue length), b (blocked processes). Per‑process details are available via pidstat -w 5, which reports voluntary switches ( cswch/s) and involuntary switches ( nvcswch/s).

When cs spikes, check r (if it exceeds the number of CPUs) and in (high interrupt rate) to decide whether the bottleneck is CPU‑bound, I/O‑bound, or interrupt‑driven.

CPU analysis workflow

Run vmstat to get overall context‑switch and interrupt rates.

Drill down with pidstat -w for per‑process voluntary and involuntary switches.

If thread‑level detail is needed, add -t (e.g., pidstat -w -t).

Correlate high r values with short‑lived processes using tools such as execsnoop or perf record/report.

For I/O‑wait spikes, combine top, dstat, and pidstat -d to locate the offending process.

CPU optimization techniques

Compile with optimization flags (e.g., gcc -O2).

Apply algorithmic improvements and asynchronous I/O to reduce blocking.

Prefer multithreading over multiprocess to lower switch overhead.

Keep hot data in CPU caches.

Use CPU affinity or binding to improve cache locality.

Adjust process niceness to lower priority of non‑critical workloads.

Limit resource usage with cgroups.

Enable NUMA‑aware memory placement and interrupt load‑balancing (e.g., irpbalance).

Memory fundamentals

Linux provides each process with a contiguous virtual address space divided into kernel and user regions. Physical pages are allocated on first access (page‑fault). The user‑space layout consists of five segments: read‑only, data, heap, mmap (file‑mapped), and stack. The MMU and multi‑level page tables translate virtual to physical addresses.

Allocation and reclamation

brk()

allocates small blocks (< 128 KB) by moving the heap top; freed memory remains cached. mmap() allocates large blocks (> 128 KB) as separate memory‑mapped regions; freed memory is returned to the kernel.

Frequent allocations increase page‑fault overhead and can cause fragmentation. The kernel reclaims memory via LRU cache eviction, swapping, and OOM killing. Example to lower a process’s OOM score:

echo -16 > /proc/$(pidof myapp)/oom_adj

Memory monitoring commands

free

– overall memory and swap. top / ps – per‑process VIRT, RES, SHR, %MEM. pidstat -r – page‑fault rates, VSZ, RSS. memleak (from BCC) – tracks allocations that are never freed; example output shows a leak in a fibonacci function.

Swap and NUMA

When memory is scarce, anonymous pages are swapped out. The aggressiveness is controlled by /proc/sys/vm/swappiness (0‑100). Three zone thresholds ( pages_min, pages_low, pages_high) guide the kswapd daemon.

In NUMA systems each node has local memory. Remote nodes can satisfy memory pressure, or the kernel can reclaim locally. Show node distribution with numactl --hardware. Control local‑only reclamation with /proc/sys/vm/zone_reclaim_mode.

Memory analysis workflow

Collect high‑level metrics with free, top, vmstat, and pidstat.

Identify the symptom (high load, high iowait, many context switches, excessive swap).

Zoom in with targeted tools ( pidstat -d, pidstat -w, perf, strace) to locate the offending process or kernel path.

Apply the appropriate mitigation (code change, configuration tweak, cgroup limit, NUMA binding, cache tuning).

Performance tools mapping

Average‑load case: uptime → mpstat / pidstat to pinpoint the high‑load process.

Context‑switch case: vmstat → pidstat -w (voluntary vs involuntary) → pidstat -w -t for thread‑level view.

High‑CPU‑process case: top to locate the process, then perf top to find the hot function.

System‑wide high‑CPU without a visible culprit: examine r column in vmstat, use pidstat -w and execsnoop to catch short‑lived processes.

Uninterruptible‑process / zombie case: use top to see D/Z states, then pstree to find parent processes, and finally perf record -d / perf report to trace kernel calls such as sys_read() or blkdev_direct_IO.

Soft‑interrupt case: inspect /proc/softirqs, correlate with sar and tcpdump to identify network‑level attacks.

CPU performance indicators

CPU usage: user (%usr), system (%sy), iowait, soft/hard interrupt percentages, steal/guest for virtualized environments.

Average load: ideally equals the number of logical CPUs; higher values indicate overload.

Process context switches: voluntary vs non‑voluntary; excessive switches waste CPU cycles.

CPU cache hit rate: higher hit rates improve performance; L1/L2 are per‑core, L3 is shared.

Memory performance indicators

System memory: total, used, free, buffers, cache, swap.

Per‑process metrics: VIRT, RES, SHR, %MEM.

Page‑fault rates: minor (in‑memory) vs major (swap‑involved).

Cache hit rate: proportion of reads satisfied from cache.

NUMA node pressure and zone thresholds.

Memory optimization practices

Disable swap or lower swappiness when possible.

Use memory pools or HugePages to reduce allocation churn.

Cache frequently accessed data in‑process or via external caches (e.g., Redis).

Apply cgroups to bound memory usage of noisy processes.

Adjust oom_score_adj for critical services.

Typical command snippets

# Create and enable an 8 GiB swap file
fallocate -l 8G /mnt/swapfile
chmod 600 /mnt/swapfile
mkswap /mnt/swapfile
swapon /mnt/swapfile

# Monitor I/O with dstat
dstat 1 10

# Trace a running process
strace -p $(pgrep app)

# Record kernel events for a process
perf record -d -p 1234
perf report