Understanding Linux Tracing Systems: Data Sources, Collection Mechanisms, and Front‑ends

Linux offers many tracing utilities at both kernel and user levels, including strace, ltrace, kprobes, tracepoints, uprobes, ftrace, perf, eBPF, sysdig, SystemTap and LTTng. This translation of Julia Evans' "Linux tracing systems & how they fit together" provides a concise overview of these tools and their relationships.

The author groups tracing into three layers: data sources (e.g., kprobes, uprobes, USDT/dtrace probes, kernel tracepoints, lttng‑ust), collection mechanisms (ftrace, perf_events, eBPF, sysdig, SystemTap, LTTng) and front‑ends that present the data interactively.

Data sources: kprobes – dynamic kernel instruction instrumentation. uprobes – user‑space function instrumentation (e.g., malloc ). USDT/dtrace probes – static user‑space trace points. Kernel tracepoints – compile‑time inserted points with minimal overhead. lttng‑ust – user‑space tracing library.

Collection mechanisms: ftrace – kernel tracing filesystem ( /sys/kernel/debug/tracing/ ). perf_events – system call that writes events to a ring buffer. eBPF – programmable in‑kernel programs attached to probes, exposing data via maps. sysdig – kernel module that records system calls and other events. SystemTap – compiles scripts into kernel modules that attach to probes. LTTng – low‑overhead tracing framework with user‑space and kernel components.

Front‑ends: perf – simple command‑line front‑end (e.g., perf trace ). ftrace front‑ends such as trace‑cmd , Catapult, KernelShark. eBPF front‑end bcc – Python‑driven framework for writing and loading eBPF programs. LTTng & SystemTap front‑ends – tools like Trace Compass that consume CTF data.

Example kprobe command:

$ sudo ./kprobe 'p:myopen do_sys_open filename=+0(%si):string'

Example uprobe command capturing bash readline input:

[email protected]~
/c/perf
-tools> sudo ./bin/uprobe 'r:bash:readline +0($retval):string'
Tracing uprobe readline (r:readline /bin/bash:0x9a520 +0($retval):string). Ctrl‑C to end.
            bash-10482 [002] d... 1061.417373: readline: (0x42176e <- 0x49a520) arg1="hi"
            ...

For kernel tracepoints, the macro TRACE_EVENT defines a trace point, e.g.:

TRACE_EVENT(udp_fail_queue_rcv_skb,
           TP_PROTO(int rc, struct sock *sk),
           TP_ARGS(rc, sk),
           TP_STRUCT__entry(
                __field(int, rc)
                __field(__u16, lport)
           ),
...)

Recommendations: on modern kernels (≥4.9) start with eBPF/bcc; for quick low‑overhead tracing use perf trace ; kprobes are useful for ad‑hoc kernel probing; ftrace is powerful but harder to use directly.