Master Linux Performance Tuning with Perf: From Basics to Advanced Profiling

Perf Overview

Perf is a performance‑diagnostic tool shipped with the Linux kernel, maintained by the kernel community. It can collect hardware (PMU), software, and tracepoint events for both user‑space applications and kernel code, making it a versatile performance‑analysis suite.

Background Knowledge

Understanding cache behavior, pipeline, superscalar execution, out‑of‑order execution, and branch prediction is essential because these hardware features directly affect application performance. Missed cache lines, pipeline stalls, or mispredicted branches can cause significant slow‑downs.

Hardware Features

Cache – A small, fast SRAM that stores frequently accessed data to avoid costly main‑memory accesses.

Pipeline & Superscalar – Processors execute multiple instructions per clock by overlapping stages; superscalar designs issue several instructions each cycle.

Out‑of‑order Execution – Allows independent instructions to run ahead of earlier ones, improving utilization when there are no data dependencies.

Branch Prediction – Predicts the outcome of conditional jumps to keep the pipeline full; mispredictions incur penalties.

Tracepoints

Tracepoints are static hooks in the kernel source that can be enabled to emit events when specific code paths execute. Perf can listen to these events, allowing developers to observe kernel‑level behavior such as slab allocations or scheduler switches.

Basic Perf Usage

perf list – Shows all available events (hardware, software, tracepoints). $ perf list perf stat – Provides a quick statistical summary of a program’s execution. $ perf stat ./t1 Typical output includes task‑clock, context‑switches, CPU migrations, page‑faults, cycles, instructions, IPC, cache references/misses, and branch statistics.

perf top – Real‑time view of the hottest functions or processes. $ perf top perf record / perf report – Records detailed sampling data and displays it, optionally with a call‑graph.

$ perf record -e cpu-clock -g ./t1
perf report

Example Programs

Program t1 – Demonstrates a CPU‑bound workload with two functions (foo1 calls longa() 100 times, foo2 calls it 10 times).

void longa(){int i,j;for(i=0;i<1000000;i++)j=i;}
void foo1(){int i;for(i=0;i<100;i++) longa();}
void foo2(){int i;for(i=0;i<10;i++) longa();}
int main(){foo1(); foo2();}

Perf stat shows a high task‑clock, confirming the program is CPU‑bound. Using perf record -g reveals that 91 % of the time is spent in foo1(), because it invokes longa() many times.

Program t2 – An infinite loop ( while(1) i++;) used to illustrate perf top identifying a hog process. while(1) i++; Program t3 – Shows the impact of branch‑target‑buffer (BTB) overflow on a Pentium 4 processor.

void foo(){int i,j;for(i=0;i<10;i++) j+=2;}
int main(){int i;for(i=0;i<100000000;i++) foo(); return 0;}

Running perf stat on the original version (20‑iteration loop) yields ~25 % branch‑misses, while the modified version (10‑iteration loop) reduces branch‑misses to ~0.003 %.

Conclusion

Perf provides a powerful set of commands for both high‑level statistical overviews and fine‑grained, function‑level profiling. By combining hardware counters, tracepoints, and call‑graph analysis, developers can locate hot spots, understand cache and branch behavior, and make informed optimization decisions.