Mastering Linux Performance with ftrace: Theory, Setup, and Real‑World Use Cases

1. ftrace Overview

ftrace (Function Trace) is a kernel‑included tracing tool first introduced in Linux 2.6.27 by Steven Rostedt in 2008. Originally a simple function logger, it has grown into a plugin‑based framework that can record kernel function calls, execution times, scheduling events, and interrupt handling, helping developers diagnose performance problems and crashes.

2. ftrace Features

2.1 Diverse Tracers

ftrace provides many tracer types, each acting like a specialised scout:

Function tracer : records every function entry and exit, useful for understanding call order, e.g., tracing a file‑system operation from vfs_open to vfs_read and vfs_close.

Function graph tracer : visualises call nesting, allowing developers to see the hierarchy of network‑stack processing.

Schedule switch tracer : logs process switches and their reasons, helping identify excessive context‑switch overhead.

Wakeup tracer : measures scheduling latency for real‑time tasks.

Irqsoff tracer : records periods when interrupts are disabled, exposing potential latency sources.

Additional tracers include Branch, Hardware branch, Initcall, Mmiotrace, Power, Sysprof, Kernel memory, Workqueue statistical, and Event tracers, each targeting specific analysis scenarios.

2.2 Light‑weight and Stable Design

ftrace uses static code instrumentation: the compiler inserts a bl mcount jump at each function entry when CONFIG_FUNCTION_TRACER is enabled. This "probe" incurs minimal overhead compared with dynamic tracing tools. Because it does not expose a complex scripting interface, the risk of user‑induced kernel crashes is low, making it more reliable than heavyweight tools such as SystemTap.

2.3 Convenient Debugfs Interface

All ftrace controls are exposed through the debugfs filesystem. After mounting debugfs (e.g., mount -t debugfs none /sys/kernel/debug), the directory /sys/kernel/debug/tracing contains files such as: available_tracers: lists all supported tracer names. current_tracer: selects the active tracer (e.g., echo function > current_tracer). tracing_on: enables (write 1) or disables (write 0) tracing. trace: the human‑readable output of the selected tracer. set_ftrace_filter and set_ftrace_pid: filter by function name or PID to reduce data volume.

3. How ftrace Works

3.1 Instrumentation Techniques

Two mechanisms are employed:

Static instrumentation : during kernel compilation, the -pg option inserts a jump to mcount at every function entry. When tracing is active, mcount calls the registered handler to record the event.

Dynamic instrumentation : with CONFIG_DYNAMIC_FTRACE, the jump can be replaced by a nop at boot, eliminating overhead. When tracing is needed, the kernel swaps the nop back to the jump for selected functions only.

3.2 Data Recording

Recorded events are written to a ring buffer, a fixed‑size circular storage that always contains the most recent data. Users read the buffer via the trace file, enabling real‑time analysis of function durations, parameters, and return values.

4. Using ftrace

4.1 Kernel Configuration

Enable the following options before building the kernel: CONFIG_FTRACE – core switch. CONFIG_FUNCTION_TRACER – basic function tracing. CONFIG_FUNCTION_GRAPH_TRACER – call‑graph visualisation. CONFIG_DYNAMIC_FTRACE – runtime enable/disable of specific functions.

Additional options such as CONFIG_IRQSOFF_TRACER and CONFIG_SCHED_TRACER can be enabled as needed.

4.2 Mounting debugfs

Check whether debugfs is already mounted: mount | grep debugfs If not, mount it: mount -t debugfs none /sys/kernel/debug Verify the presence of the tracing directory under /sys/kernel/debug.

4.3 Common Files and Commands

View available tracers: cat /sys/kernel/debug/tracing/available_tracers Select a tracer:

echo function_graph > /sys/kernel/debug/tracing/current_tracer

Enable tracing:

echo 1 > /sys/kernel/debug/tracing/tracing_on

Disable tracing:

echo 0 > /sys/kernel/debug/tracing/tracing_on

Read the trace output: cat /sys/kernel/debug/tracing/trace Filter by function:

echo vfs_read > /sys/kernel/debug/tracing/set_ftrace_filter

Filter by PID:

echo 1234 > /sys/kernel/debug/tracing/set_ftrace_pid

5. Real‑World Cases

5.1 Performance Optimisation

When a file‑system module exhibited slow reads/writes, the developer:

Mounted debugfs and switched to the function_graph tracer.

Restricted tracing to vfs_read and vfs_write via set_graph_function.

Ran typical workload, stopped tracing, and examined trace.

The trace showed a long‑running page_cache_sync_readahead and its downstream ra_submit calls. After optimising ra_submit to avoid unnecessary I/O, subsequent traces showed markedly reduced durations for vfs_read / vfs_write, confirming the performance gain.

5.2 Fault Diagnosis

In a production system suffering from high response latency, the team suspected interrupt disabling. They enabled the irqsoff tracer, captured a latency of 12345 µs, and inspected the call stack, which pointed to some_function_3. The function unnecessarily disabled interrupts while performing heavy data processing. After shortening the critical section and improving the algorithm, latency disappeared and normal response times were restored.

6. Comparison with Other Tracing Tools

6.1 ftrace vs. SystemTap

SystemTap offers a powerful scripting language for custom probes, enabling fine‑grained analysis but at the cost of complexity and a higher risk of kernel instability if scripts are incorrect. ftrace’s static‑instrumentation approach is simpler, more stable, and incurs negligible overhead, making it preferable for routine kernel debugging in production environments.

6.2 ftrace vs. LTTng

LTTng uses a binary interface and separate ring buffers per tracer, which yields efficient storage and powerful post‑processing tools for embedded systems. ftrace, by contrast, provides an ASCII interface and a single shared ring buffer, offering immediate readability and ease of use for kernel developers. The choice depends on whether quick, on‑the‑fly inspection (ftrace) or deep, tool‑driven analysis (LTTng) is required.