Unlock Performance Bottlenecks with Flame Graphs: A Practical Guide

Introduction

The evolution of tools drives productivity; using the right tool speeds up work and problem diagnosis. While shell commands excel at aggregating text data, they struggle with relative values and multidimensional analysis, prompting the need for visual tools such as graphs.

Overview

Motivation

When troubleshooting performance, we often dump thread stacks and run commands like

grep --no-group-separator -A 1 java.lang.Thread.State jstack.log | awk 'NR%2==0' | sort | uniq -c | sort -nr

to see what most threads are doing. The frequency of stack appearances approximates the time spent in each call, similar to counting how often a billboard ad appears in random photos.

2444  at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1200)
   1587  at sun.misc.Unsafe.park(Native Method)
    795  at java.security.Provider.getService(Provider.java:1035)
    293  at java.lang.Object.wait(Native Method)
    292  at java.lang.Thread.sleep(Native Method)
     73  at org.apache.logging.log4j.core.layout.TextEncoderHelper.copyDataToDestination(TextEncoderHelper.java:61)
     71  at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
     70  at java.lang.Class.forName0(Native Method)
     54  at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.checkRollover(RollingFileManager.java:217)

Shell scripts become cumbersome for deeper analysis because thread stacks have call‑chain and frequency dimensions. Brendan Gregg introduced flame graphs to address this.

Introduction

Flame graphs, named for their flame‑like appearance, are interactive SVGs. Clicking a block expands its call chain upward, and hovering shows detailed information. A typical flame graph consists of stacked rectangles of varying width and color, each labeled with a function name.

https://github.com/brendangregg/FlameGraph

Features

From bottom to top you can trace a unique call chain; each block’s parent is the block directly below it.

Blocks with the same parent are ordered alphabetically from left to right.

The text on a block shows the function name; the number in parentheses indicates how many times that call appears, and the block’s width represents its percentage of the total stack samples.

Colors have no semantic meaning; they are only visual aids.

Analysis

When reading a flame graph, focus on block widths because they reflect sample frequency and thus time consumption. Wide blocks in the middle may not be problematic if their children are evenly distributed. The most critical blocks are flat‑topped rectangles at the top—these have no children, consume a lot of time, and are likely the performance culprits.

Use Cases

Loop analysis: large or infinite loops create flat tops near the top of the graph, indicating frequent thread‑stack switches.

I/O bottleneck / lock analysis: synchronous calls that block on I/O or locks appear as long‑lasting blocks where threads wait, clearly visible in the flame graph.

Inverted flame graph: merging identical top‑level calls across many branches reveals the total cost of a shared function, helping assess optimization impact.

Implementation

Generation Tool

Brendan Gregg provides a Perl script flamegraph.pl that generates flame graphs. It accepts parameters to adjust colors, size, and other visual aspects.

The script expects input in the form:

a;b;c 12<br/>a;d 3<br/>b;c 3<br/>z;d 5<br/>a;c;e 3

Each line lists a call chain separated by semicolons, followed by the number of occurrences.

Data Preparation

Various helper scripts convert raw dumps to the required format: stackcollapse-perf.pl for perf output, stackcollapse-jstack.pl for jstack output, and stackcollapse-gdb.pl for gdb stacks. A simple shell pipeline can also process jstack data:

grep -v -P '.+prio=d+ os_prio=d+' | grep -v -E 'locked <' | awk '{if ($0==""){print $0}else{printf"%s;",$0}}' | sort | uniq -c | awk '{a=$1;$1="";print $0,a}'

Conclusion

Flame graphs provide a powerful visual method for diagnosing performance issues, adding another tool to a developer’s toolbox. The author plans to create a series dedicated to useful development tools.