Mastering Linux Performance with ftrace: Theory, Features, and Real‑World Use Cases

1. Overview of ftrace

ftrace (Function Trace) is a powerful tracing tool built into the Linux kernel. It was created in 2008 by Steven Rostedt and first appeared in kernel 2.6.27. Initially a simple function logger, it has evolved into a plugin‑based framework supporting many tracer types, making it essential for kernel development and debugging.

2. ftrace Features

2.1 Diverse Tracers

ftrace provides several built‑in tracers, each focusing on a different aspect of kernel behavior:

Function tracer : records every kernel function call and its timing.

Function graph tracer : visualises call hierarchies and nesting.

Schedule switch tracer : logs process scheduling events and reasons.

Wakeup tracer : measures scheduling latency for real‑time tasks.

Irqsoff tracer : tracks periods when interrupts are disabled.

Additional tracers include Branch, Hardware branch, Initcall, Mmiotrace, Power, Sysprof, Kernel memory, Workqueue statistical, and Event tracers.

2.2 Light‑weight and Stable Design

ftrace uses static code instrumentation: during kernel compilation a -pg option inserts a probe at each function entry. This adds minimal overhead compared to dynamic tracing tools and keeps the impact on production systems negligible. Because it does not expose complex user‑defined interfaces, the risk of kernel crashes from misuse is low, giving it higher stability than tools like SystemTap.

2.3 Simple Debugfs Interface

All ftrace controls are exposed through the debugfs filesystem. After mounting debugfs, the directory /sys/kernel/debug/tracing contains files such as: available_tracers – lists all supported tracers. current_tracer – selects the active tracer (e.g., function, function_graph). tracing_on – enables (write 1) or disables (write 0) tracing. set_ftrace_filter – filters functions to trace. set_ftrace_pid – filters by process ID.

These files can be read or written with standard shell commands ( cat, echo), allowing quick configuration without complex command‑line syntax.

3. How ftrace Works

3.1 Instrumentation Techniques

Two techniques are used:

Static instrumentation : the compiler inserts a bl mcount jump at each function entry. When the function runs, control passes to mcount, which checks the registered handler ( ftrace_trace_function) and records data if tracing is active.

Dynamic instrumentation : at boot the inserted jumps are replaced by nop instructions, eliminating overhead. When tracing is needed, the kernel swaps the nop back to the jump for selected functions only, providing on‑demand tracing.

3.2 Data Recording

Recorded events are stored in a ring buffer, a fixed‑size circular queue. New entries overwrite the oldest ones, ensuring the buffer always contains the most recent data while preventing overflow. Users read the buffer via the trace file.

4. Using ftrace

4.1 Kernel Configuration

Enable the following kernel options before building: CONFIG_FTRACE – core switch. CONFIG_FUNCTION_TRACER – basic function tracing. CONFIG_FUNCTION_GRAPH_TRACER – call‑graph visualisation. CONFIG_DYNAMIC_FTRACE – runtime enable/disable.

Other optional options include CONFIG_IRQSOFF_TRACER, CONFIG_SCHED_TRACER, etc.

4.2 Mount debugfs

Check if debugfs is already mounted: mount | grep debugfs If not, mount it: mount -t debugfs none /sys/kernel/debug Verify the presence of the tracing directory under /sys/kernel/debug.

4.3 Common Files and Commands

Enable tracing: echo 1 > /sys/kernel/debug/tracing/tracing_on Disable tracing: echo 0 > /sys/kernel/debug/tracing/tracing_on Select tracer:

echo function_graph > /sys/kernel/debug/tracing/current_tracer

List available tracers: cat /sys/kernel/debug/tracing/available_tracers Filter functions (e.g., vfs_read):

echo vfs_read > /sys/kernel/debug/tracing/set_ftrace_filter

Filter by PID (e.g., 1234):

echo 1234 > /sys/kernel/debug/tracing/set_ftrace_pid

View trace output:

cat /sys/kernel/debug/tracing/trace

5. Practical Cases

5.1 Performance Optimisation

Scenario: a kernel module handling heavy file I/O shows degraded throughput. Using the function‑graph tracer on vfs_read and vfs_write, the trace revealed that page_cache_sync_readahead and its downstream calls consumed excessive time. After optimising ra_submit to avoid unnecessary disk I/O, subsequent traces showed markedly reduced execution times and improved read/write performance.

5.2 Fault Diagnosis

Scenario: production servers experience intermittent response latency. The team suspected interrupt disabling. By enabling the irqsoff tracer and reproducing the issue, the trace displayed a worst‑case IRQ‑off latency of 12345 µs and a call stack ending in some_function_3. Investigation showed that this function disabled interrupts for too long while processing data. Refactoring the code to minimise the critical section eliminated the latency spikes and restored normal response times.

6. Comparison with Other Tools

6.1 ftrace vs. SystemTap

SystemTap offers a powerful scripting language for custom probes, enabling fine‑grained analysis but at the cost of complexity and a higher risk of kernel instability if scripts contain errors. ftrace, by contrast, relies on static instrumentation, provides a small, stable code base, and incurs virtually no runtime overhead, making it safer for production environments.

6.2 ftrace vs. LTTng

LTTng uses a binary interface and separate ring buffers per tracer, which allows efficient storage and advanced post‑processing tools but requires dedicated analysis utilities. ftrace outputs plain ASCII data that can be inspected directly with cat. While LTTng’s design reduces buffer contention in heavy‑load scenarios, ftrace’s unified ring buffer simplifies usage for quick, on‑the‑fly debugging.