A Comprehensive Guide to Linux Performance Optimization

Overview

Linux performance optimization focuses on two core indicators—throughput and latency. The article explains that performance problems arise when system resources hit a bottleneck and request handling is too slow, and defines performance analysis as the process of locating and mitigating those bottlenecks.

Key Metrics

Average Load : average number of runnable or uninterruptible processes; not directly comparable to CPU usage.

CPU Usage : user, system, iowait, softirq, steal, and guest percentages.

Context Switches : voluntary (cs) and involuntary (nvcswch) switches per second.

Memory : virtual vs. physical allocation, page faults, and the layout of the virtual address space (code, data, heap, mmap, stack).

Swap and Swappiness : when memory pressure forces pages to disk.

Performance Analysis Workflow

Identify the symptom (high load, high CPU, excessive iowait, many uninterruptible or zombie processes).

Collect baseline data with broad‑scope tools such as top, free, vmstat, and pidstat.

Drill down to the offending process using pidstat -w (context switches), pidstat -u (CPU), or pidstat -r (memory).

When generic tools cannot pinpoint the cause, use tracing tools like perf top, perf record / perf report, strace, or perf record -d to examine call stacks.

Validate hypotheses with targeted commands (e.g., watch cat /proc/interrupts, pstree, dstat).

CPU‑Centric Cases

Examples include a scenario where sysbench spikes CPU usage, but pidstat shows low context‑switch counts, leading to the discovery that the benchmark stresses threads. Another case shows high system CPU without a visible high‑CPU process; the analysis reveals many processes stuck in the Running state and a surge in interrupt counts, prompting inspection of /proc/softirqs and perf to locate the root cause.

Memory‑Centric Cases

The guide explains how malloc uses brk() for small allocations and mmap() for large ones, why frequent allocations cause page‑fault overhead, and how to detect memory leaks with memleak from the BCC tools. It also shows how to interpret free, top, and pidstat -r output to differentiate between used, cached, and buffered memory.

Swap Analysis

When swap usage rises despite ample free memory, the article advises checking /proc/zoneinfo thresholds (pages_min, pages_low, pages_high) and the swappiness setting. It demonstrates creating a swap file with fallocate, enabling it with mkswap / swapon, and then monitoring swap impact with sar -S and cachetop.

Toolset Summary

vmstat

– system‑wide CPU, memory, I/O, and interrupt statistics. pidstat – per‑process CPU, memory, I/O, and context‑switch metrics. perf – low‑level profiling of functions and call stacks. dstat – combined CPU and I/O monitoring for quick correlation. strace – system‑call tracing for debugging I/O or O_DIRECT usage.

BCC tools ( memleak, cachestat, cachetop) – deep insight into memory allocation and cache behavior.

Optimization Strategies

Application‑level: enable compiler optimizations (e.g., gcc -O2), improve algorithms, use asynchronous I/O, replace processes with threads, and leverage caching.

System‑level: bind processes to CPUs, adjust nice values, configure cgroups limits, enable NUMA‑aware scheduling, and balance interrupts with irqpbalance.

Memory‑level: prefer memory pools or HugePages, reduce dynamic allocations, and tune swappiness to limit swap usage.

Practical Commands

vmstat 5

pidstat -w 5 10

pidstat -u 1 10

pidstat -r 1 10

perf top -g -p <pid>

strace -p <pid>

sudo docker run --privileged --name=app -itd feisky/app:iowait

Images