Understanding AI Flame Graphs: Insights from Brendan Gregg

Intel has created an AI Flame Graph, a visual profiling tool modeled after the classic CPU flame graph, to show AI accelerator or GPU hardware profiles together with the complete software stack. The first preview runs on Intel Data Center GPU Max (formerly Ponte Vecchio) in the Intel Tiber AI Cloud.

Simple SYCL Matrix‑Multiplication Example

The flame graph visualizes three SYCL implementations of matrix multiplication. The unoptimized multiply_basic() dominates with 72% stall samples, while multiply_local_access() and multiply_local_access_and_tiling() occupy 21% and 6% respectively, illustrating how added optimizations shrink the flame‑graph towers.

Instruction‑Offset Analysis

Unlike earlier AI profilers that only trace CPU stacks, this tool also captures instruction offsets on accelerators without expensive binary instrumentation, aiming for a CPU‑flame‑graph‑like experience: easy to use, negligible overhead, production‑safe, and source‑code‑centric.

What Is a Flame Graph?

Invented in 2011, a flame graph visualizes sampled call‑stack data with the x‑axis proportional to resource cost. It quickly highlights the widest rectangles, which correspond to the most expensive code paths, reducing hours of log parsing to seconds.

Search Samples

The built‑in search highlights frames containing a term (e.g., “sbid”) and shows the percentage of samples (78.4% in the example). Samples are based on EU stall profiling, which measures time spent stalled rather than wall‑clock time.

Who Will Use It?

At a recent GoLang conference, over 200 attendees raised their hands to indicate they use CPU flame graphs; the author expects AI developers to adopt AI flame graphs as a daily debugging tool to cut compute costs.

Why AI Analysis Is Hard

AI workloads run on diverse runtimes, frameworks, and drivers, many of which lack standard file‑system presence or /proc entries. Capturing instruction streams from GPU/AI accelerators often requires high‑overhead, hardware‑specific debugger interfaces, making stack correlation difficult.

AI Developers’ Reaction

Developers are initially confused but soon appreciate seeing the full stack—including hardware—visible in a single view, similar to the surprise many felt when first using CPU flame graphs.

PyTorch Support

The first PyTorch AI flame graph shows an Llama 2 7B model running with Intel Extensions for PyTorch. Pink frames represent Python code, while the dominant stall samples (27% of the profile) come from a GEMM kernel. This milestone suggests broader framework support is feasible.

First Release: Difficulty and Moderate Overhead

Some workloads are easy to profile; others, like PyTorch, require weeks of OS‑level work (e.g., enabling frame pointers). The team expects a year to upstream many required patches, after which overhead will drop and coverage will expand.

Usability

Initially the AI flame graph will be a preview feature in Intel Tiber AI Cloud for the Data Center GPU Max series. Wider hardware support, open‑source release, and broader availability depend on other Intel teams.

Conclusion

Even a few‑percent performance improvement in AI data centers can save massive electricity, water, and cost globally. If AI flame graphs achieve adoption similar to CPU flame graphs, 10‑plus‑percent gains could become common, with occasional 50%+ wins, despite current challenges in software preparation.