Introduction to ftrace and Its Practical Usage for Linux Kernel Tracing

Although many have heard of ftrace, it is often not used in practice; this guide introduces ftrace, its history, and why it is essential for diagnosing kernel issues.

1. ftrace Overview

ftrace (Function Tracer) is an internal Linux kernel tracing framework that helps developers understand kernel behavior, debug latency, and performance problems. It consists of a framework core that manages various tracers and a set of tracer plugins such as function, function_graph, irqsoff, and preemptoff.

1.1 What is ftrace?

ftrace records kernel events like scheduling, interrupts, and function calls, providing insight into execution order, timing, and stack usage. It originated in 2008 and has evolved into a flexible plugin‑based tracing system.

1.2 How ftrace Works

Probes are inserted into kernel functions; their data is written to a ring buffer accessible via tracefs or debugfs . The core manages tracer registration, configuration files, and buffer control.

1.3 Tracer Types

Common tracers include function (simple call logging), function_graph (call hierarchy), nop (disable tracing), irqsoff (interrupt‑off latency), preemptoff (preempt‑disable latency), and many event‑based tracers.

2. Using ftrace

Mount debugfs or tracefs (usually at /sys/kernel/debug/tracing or /sys/kernel/tracing ) and ensure kernel options enable ftrace.

mkdir /debug; mount -t debugfs nodev /debug

echo nop > current_tracer

echo function_graph > current_tracer

Select a tracer by writing its name to current_tracer , enable tracing with echo 1 > tracing_on , and view results in trace or trace_pipe .

2.1 Function Tracer Example

echo function > current_tracer

cat trace | head -n 15

Filter specific functions:

echo schedule > set_ftrace_filter

Exclude functions:

echo '!schedule' > set_ftrace_filter

2.2 Function Graph Tracer

echo function_graph > current_tracer

cat trace | head -n 15

Trace a single function’s call stack:

echo kfree > set_graph_function

2.3 Event Tracing

Enable an event, e.g., sched_wakeup :

echo 1 > events/sched/sched_wakeup/enable

List available events with ls events/ and explore specific event formats.

2.4 trace‑cmd Frontend

Use trace‑cmd record -e sched to record and trace‑cmd report to view filtered output.

3. ftrace Probing Techniques

3.1 Static Probes

Compiled into the kernel, static tracepoints are always present but can be enabled/disabled at runtime.

3.2 Dynamic Probes

GCC inserts stubs at function entry; ftrace replaces them with tracing code when activated, minimizing overhead when disabled.

4. Practical Usage

4.1 Installation

Enable CONFIG_DEBUG_FS=y and mount debugfs and tracefs as shown above.

4.2 Basic Workflow

Select a tracer via current_tracer .

Optionally set filters ( set_ftrace_filter , set_ftrace_notrace ).

Enable tracing with echo 1 > tracing_on .

Run the workload.

Disable tracing with echo 0 > tracing_on or reset with echo nop > current_tracer .

4.3 Analyzing Results

Read human‑readable output from trace or stream live data from trace_pipe . Information includes task name, PID, CPU, timestamp, and function name, useful for locating latency, call order, and performance bottlenecks.

5. Application Scenarios

Performance optimization – identify slow kernel functions.

Fault diagnosis – capture call stacks before crashes.

Kernel development – verify module behavior and scheduling.

6. Comparison with Other Tools

ftrace vs. perf : ftrace provides detailed function‑level tracing, while perf focuses on statistical sampling. ftrace vs. SystemTap: SystemTap offers flexible scripting at the cost of higher learning curve; ftrace is simpler, using built‑in tracers.

7. Real‑World Case Study

A step‑by‑step investigation of I/O latency using iosnoop , tpoint , funccount , funcslower , and kprobe demonstrates how ftrace can pinpoint mis‑configured readahead settings and resolve performance issues.