How to Use ftrace Event Tracing to Debug Android Process Scheduling

1. ftrace Basic Principles

ftrace is a Linux kernel tracing framework that registers various tracers to a central core. Each tracer provides a probe point, a probe function that writes data to a ring buffer, and a debugfs interface for user‑space control. The ring buffer and debugfs management are handled by the framework.

2. ftrace Event Tracing Mechanism

Event tracing builds on tracepoints. A tracepoint defines a static probe point in the kernel source; when enabled, the kernel calls a generated probe function that records data into the ring buffer. The generic mechanism is shared by other tracers such as function tracer and kprobe.

3. Tracepoint Implementation

Two key macros in <linux/tracepoint.h> drive tracepoint creation: DECLARE_TRACE declares the probe point and associated data structures. DEFINE_TRACE defines a string table entry and a struct tracepoint placed in a special linker section.

The macro TRACE_EVENT expands to a probe point, a probe function, and the format used to print the event.

4. Event Tracing Implementation Steps

Define an event with TRACE_EVENT, specifying the prototype and print format.

Enable the event by writing 1 to /sys/kernel/debug/tracing/events/<event_name>/enable.

Optionally filter the event (see section 5).

Read the recorded data from the tracefs file system (e.g., cat /sys/kernel/debug/tracing/trace).

5. Using ftrace on Android

5.1 Android systrace tag list

Run atrace --list to display all supported systrace tags.

5.2 User‑space trace_marker

Applications can write custom markers to the kernel trace using the trace_marker interface. The Java API provides Trace.traceBegin(tag, name) / Trace.traceEnd(tag), while the native API offers ATRACE_BEGIN(name) / ATRACE_END(). The marker can be enabled or disabled by writing 1 or 0 to options/markers.

6. Android Process State Transitions

The Android runtime defines four main states: Runnable → Running, Running → Runnable, Running → Blocked, and Blocked → Runnable. Understanding these transitions helps map scheduling events to observable behavior.

7. Scheduling‑related Events and Filtering

Kernel scheduling events are located under events/sched/. Typical events useful for thread‑level analysis include sched_switch, sched_wakeup, etc. Filters can be applied using simple expressions such as comm == "myprocess" or pid == 1234. The filter syntax supports numeric operators (==, !=, <, <=, >, >=, &) and string operators (==, !=, ~) with up to 16 conditions.

8. Practical Example: Tracing a Low‑Probability Camera Crash

A real‑world issue involved occasional camera preview stalls and crashes during mode switches. The problem was reproduced by:

Increasing the ftrace buffer size.

Filtering events to the APSRoutine process (identified by its comm name).

Enabling trace_marker to capture user‑space actions.

The workflow captured only the relevant scheduling events, reducing log volume and allowing the developers to pinpoint the abnormal scheduling behavior associated with AlgoInterface::init.

9. Conclusion

The article covered the ftrace core architecture, event‑tracing macros, Android‑specific usage via systrace and trace_marker, and a concrete debugging scenario. Readers can now build small utilities to trace arbitrary Java or native processes on Android and extend them for broader performance analysis.