Comprehensive Guide to Linux Kernel ftrace: Concepts, Usage, and Case Studies

ftrace is a tracing tool built into the Linux kernel that helps developers analyze and debug performance issues by tracking function calls, system calls, interrupt events, timer events, and more, generating detailed trace data.

One of the most common uses of ftrace is event tracing; the kernel contains hundreds of static tracepoints that can be enabled via the Tracefs filesystem to observe kernel activity.

By analyzing this data, developers can understand execution paths, function latency, and resource usage, enabling performance optimization and fault isolation. ftrace offers components such as function graph, event tracing, and dynamic tracing, making it valuable in many scenarios.

1. Introduction to ftrace

ftrace provides insight into the runtime behavior of the Linux kernel, allowing inspection of system calls and function call flows. It consists of two parts: a performance analyzer (profiler) and a tracer.

Performance Analyzer : Supplies statistics and histograms (requires CONFIG_FUNCTION_PROFILER=y).

Function performance analysis

Histograms

Tracer : Provides detailed information about traced events.

Function tracing (function)

Tracepoint tracing (tracepoint)

kprobe

uprobe

Function call graph (function_graph)

hwlat, etc.

The ftrace framework also includes front‑end tools such as perf‑tools and trace‑cmd.

2. Using ftrace

1) Enable ftrace in the kernel : Ensure the relevant kernel configuration options are set (Kernel hacking → Tracers) and enable DEBUG_FS (CONFIG_DEBUG_FS=y).

2) Mount debugfs to expose the tracing interface.

3) User‑space interface : The tracing directory is /sys/kernel/debug/tracing/ . Common files include:

tracing_on : Enable (1) or disable (0) writing to the trace buffer.

trace : Textual view of the trace buffer.

current_tracer : Select the active tracer (e.g., function, function_graph).

available_tracers : List of tracers supported by the kernel.

tracing_cpumask : Hex mask to select CPUs for tracing.

dynamic tracing : Interface for runtime filtering (requires compile‑time support).

set_ftrace_pid : Restrict tracing to a specific PID.

trace_pipe : Stream trace output like a pipe.

buffer_size_kb : Size of per‑CPU trace buffers (requires current_tracer=nop ).

set_ftrace_filter : Include only specified functions (supports a single wildcard).

set_ftrace_notrace : Exclude specified functions.

4) Tracer plugins (selected via current_tracer ):

function – tracks every function call.

function_graph – records call relationships and timings.

mmiotrace – traces MMIO accesses (e.g., Nouveau driver).

blk – block I/O tracing.

wakeup / wakeup_rt – measures latency from wake‑up to execution.

irqsoff – measures interrupt‑disable latency.

nop – disables all tracing.

5) Typical usage workflow (three steps) :

Set the tracer type (e.g., echo function > current_tracer ).

Configure tracer parameters (e.g., echo blk_update_request > set_ftrace_filter ).

Enable the tracer ( echo 1 > tracing_on ).

Function tracer example

echo 0 > tracing_on
cat trace > /dev/null
echo function > current_tracer
echo blk_update_request > set_ftrace_filter
echo 1 > tracing_on
cat trace

The output shows task PID, CPU, timestamps, and function names, allowing extraction of process information, CPU usage, system state, and timing.

Function graph example

echo function_graph > current_tracer
echo blk_update_request > set_graph_function
echo 1 > options/func_stack_trace
echo 1 > tracing_on
cat trace

The graph displays nested function calls with durations, useful for visualizing call hierarchies.

Event tracing example

Enable a tracepoint such as sched:sched_stat_runtime to monitor scheduler runtime statistics:

echo 0 > tracing_on
cat > trace
echo sched:sched_stat_runtime > set_event
echo 1 > tracing_on
cat trace

Filters can be applied using C‑like expressions (e.g., echo 'comm ~ "sh*"' > filter ).

IRQ latency tracer

echo irqsoff > current_tracer
echo 1 > tracing_on
# run workload
cat trace

The output lists the duration of interrupt‑disabled sections, helping locate sources of latency.

Trace marker

Users can write arbitrary strings to trace_marker ; ftrace records the timestamp, enabling custom markers in user programs.

#include <stdio.h>
int main(){
    int fd = open("/sys/kernel/debug/tracing/trace_marker", O_WRONLY);
    write(fd, "start program\n", 13);
    // ...
    close(fd);
    return 0;
}

3. Case Studies

3.1 Function tracing a specific process

Use set_ftrace_pid to limit tracing to a PID.

Apply set_ftrace_filter to focus on particular functions.

3.2 Function graph with stack traces

Enable options/func_stack_trace to capture call stacks for each function entry.

3.3 Event tracing with filters

Filter events by command name or PID using the filter file in the event directory.

3.4 IRQ latency measurement

Use the irqsoff tracer to measure the time spent with interrupts disabled.

3.5 Trace marker usage in applications

Write markers from user space to correlate application phases with kernel activity.

3.6 Wake‑up latency tracing

Tracer wakeup and wakeup_rt record the delay from a task being woken to its actual execution.

4. Tools for Data Collection and Visualization

trace‑cmd follows a reset → record → stop → report workflow, producing a trace.dat file that can be examined with trace‑cmd report .

kernelshark provides a graphical UI to visualize trace.dat , making it easier to explore complex trace data.

Both tools are essential for large‑scale performance investigations and debugging sessions.

Article promotion: Join the Linux kernel technology discussion group (ID: 865977150) for additional resources and recommended reading.