Profiling Software with perf and Flame Graphs: A Step‑by‑Step Guide

This guide demonstrates how to use Linux's perf utility together with Brendan Gregg's FlameGraph scripts to visualize function‑level performance data of a running MySQL server.

Installation : On CentOS 7 install perf with sudo yum install -y perf. Then clone the FlameGraph repository:

mkdir -p ~/src

cd ~/src

git clone https://github.com/brendangregg/FlameGraph

Collecting samples : Three common perf record patterns are shown. To record for a fixed 10 seconds on the MySQL process:

sudo perf record -a -F 99 -g -p $(pgrep -x mysqld) --sleep 10

To record until interrupted (Ctrl+C): sudo perf record -a -F 99 -g -p $(pgrep -x mysqld) To record the entire lifecycle of the MySQL binary:

sudo perf record -a -F 99 -g -- /sbin/mysqld \

--defaults-file=/etc/percona-server.conf.d/mysqld.cnf --user=mysql

Preparing the data : Convert the binary perf.data to a human‑readable script: sudo perf script > perf.script You can also specify input and output files with -i and -o options.

Generating the flame graph : Pipe the script through the FlameGraph utilities in a single command:

~/src/FlameGraph/stackcollapse-perf.pl perf.script | ~/src/FlameGraph/flamegraph.pl > flamegraph.svg

Open the resulting .svg in a browser to explore the stacked function calls.

Example : Profiling an INSERT … SELECT workload:

time sudo perf record -a -F 99 -g -p $(pgrep -x mysqld) \

--mysql test -e "INSERT INTO joinit SELECT NULL, uuid(), time(now()), (FLOOR(1+RAND()*60)) FROM joinit;"

After recording, generate the graph with the same pipeline as above. The flame graph highlights write_record as a hotspot, accounting for roughly 60 % of samples.

Interpretation : Hovering over functions in the SVG reveals inclusive time percentages; wide, low‑stack frames indicate hot paths. The guide links to further reading on flame‑graph interpretation.

Summary : Using perf and FlameGraph provides MySQL support engineers with deep insight into execution paths and timing, enabling targeted optimizations and effective troubleshooting.

Related links :

Interpreting Flame Graphs – https://queue.acm.org/detail.cfm?id=2927301

Flame Graphs 201 – Percona webinar https://www.percona.com/resources/webinars/flame-graphs-201

Brendan Gregg’s FlameGraph site http://www.brendangregg.com/flamegraphs.html