Unlock Linux Observability: A Practical Guide to eBPF, SystemTap, and DTrace

1. Introduction

eBPF (extended Berkeley Packet Filter) is a kernel technology that allows developers to run specific functions without modifying kernel code. The concept originates from the Berkeley Packet Filter (BPF), a network filter created at Bell Labs for capturing and filtering packets.

To meet the demand for better Linux tracing tools, eBPF drew inspiration from dtrace, a dynamic tracing tool for Solaris and BSD. Unlike dtrace, Linux lacks a comprehensive view of the running system because it is limited to system calls, library calls, and specific function frameworks.

Building on BPF, a small group of engineers extended the backend to provide dtrace‑like capabilities, leading to the birth of eBPF. It was first released in Linux 3.18 in 2014, but fully usable eBPF requires Linux 4.4 or newer.

Compared with traditional BPF, which is limited to network filtering, eBPF can be used for many scenarios, including network monitoring, security filtering, and performance analysis. eBPF lets user‑space applications package logic as bytecode that the kernel executes when certain events (hooks) occur, such as system calls or network events. The most popular toolchain for writing and debugging eBPF programs is the BPF Compiler Collection (BCC), based on LLVM and Clang.

Similar tools include SystemTap, an open‑source utility that collects runtime data by dynamically loading kernel modules, and DTrace, a dynamic tracing and analysis tool. The following table compares eBPF, SystemTap, and DTrace:

From the table, all three tools are powerful for collecting and analyzing system runtime information.

Use Cases

eBPF is a flexible and powerful kernel technology applicable to many scenarios. Common use cases include:

Network monitoring: capture packets and run custom logic to analyze traffic, alerting on anomalies.

Security filtering: block or intercept malicious traffic directly in the kernel.

Performance analysis: collect kernel performance metrics, visualize them, and identify bottlenecks.

Virtualization: gather VM performance data and perform load balancing to improve resource utilization.

In summary, eBPF can be used for network monitoring, security filtering, performance analysis, and virtualization.

2. How eBPF Works

eBPF operates in three steps: loading, compilation, and execution.

First, a user‑space application loads the eBPF program via a system call, copying the code into kernel space.

The program is then compiled (typically by Clang/LLVM) into bytecode, which the kernel JIT‑compiles into native instructions for efficient execution.

At runtime, the eBPF program attaches to a kernel hook so it runs when specific events occur, such as a network packet arriving.

Before execution, the kernel verifies the program through a safety checker to ensure it cannot destabilize or compromise the system. Once approved, the program can access kernel data structures and interact with other components.

3. Example: Packet Counting with eBPF

The following Python script uses BCC to count TCP packets in real time.

#!/usr/bin/python3
from bcc import BPF
from time import sleep
# Define eBPF program
bpf_text = """
#include <uapi/linux/ptrace.h>
BPF_HASH(stats, u32);
int count(struct pt_regs *ctx) {
    u32 key = 0;
    u64 *val, zero = 0;
    val = stats.lookup_or_init(&key, &zero);
    (*val)++;
    return 0;
}
"""
# Compile eBPF program
b = BPF(text=bpf_text, cflags=["-Wno-macro-redefined"])
# Load eBPF program
b.attach_kprobe(event="tcp_sendmsg", fn_name="count")
name = {0: "tcp_sendmsg"}
# Output statistics
while True:
    try:
        for k, v in b["stats"].items():
            print("{}: {}".format(name[k.value], v.value))
        sleep(1)
    except KeyboardInterrupt:
        exit()

This program defines a BPF_HASH map to store packet counts, attaches to the tcp_sendmsg kernel function, and prints the statistics every second.

To run the script, install BCC and its dependencies (example for Ubuntu 20.10+): sudo apt install python3-bpfcc Save the script as netstat.py, make it executable, and run:

$ chmod +x ./netstat.py
$ sudo ./netstat.py
tcp_sendmsg: 29
tcp_sendmsg: 216
...

Press Ctrl+C to stop.

4. Advanced Example: Measuring TCP Latency

The next script records timestamps on packet send and receive to compute transmission latency.

#!/usr/bin/python3
from bcc import BPF
import time
# Define eBPF program
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <net/inet_sock.h>
#include <bcc/proto.h>
struct packet_t {
    u64 ts, size;
    u32 pid;
    u32 saddr, daddr;
    u16 sport, dport;
};
BPF_HASH(packets, u64, struct packet_t);
int on_send(struct pt_regs *ctx, struct sock *sk, struct msghdr *msg, size_t size) {
    u64 id = bpf_get_current_pid_tgid();
    u32 pid = id;
    struct packet_t pkt = {};
    pkt.ts = bpf_ktime_get_ns();
    pkt.size = size;
    pkt.pid = pid;
    pkt.saddr = sk->__sk_common.skc_rcv_saddr;
    pkt.daddr = sk->__sk_common.skc_daddr;
    struct inet_sock *sockp = (struct inet_sock *)sk;
    pkt.sport = sockp->inet_sport;
    pkt.dport = sk->__sk_common.skc_dport;
    packets.update(&id, &pkt);
    return 0;
}
int on_recv(struct pt_regs *ctx, struct sock *sk) {
    u64 id = bpf_get_current_pid_tgid();
    struct packet_t *pkt = packets.lookup(&id);
    if (!pkt) { return 0; }
    u64 delta = bpf_ktime_get_ns() - pkt->ts;
    bpf_trace_printk("tcp_time: %llu.%llums, size: %llu
", delta/1000, delta%1000%100, pkt->size);
    packets.delete(&id);
    return 0;
}
"""
# Compile eBPF program
b = BPF(text=bpf_text, cflags=["-Wno-macro-redefined"])
# Register eBPF programs
b.attach_kprobe(event="tcp_sendmsg", fn_name="on_send")
b.attach_kprobe(event="tcp_v4_do_rcv", fn_name="on_recv")
print("Tracing TCP latency... Hit Ctrl-C to end.")
while True:
    try:
        (task, pid, cpu, flags, ts, msg) = b.trace_fields()
        print("%-18.9f %-16s %-6d %s" % (ts, task, pid, msg))
    except KeyboardInterrupt:
        exit()

This program captures timestamps on send and receive events, computes the delta, and prints the latency.

5. BCC Tool Suite

BCC (BPF Compiler Collection) provides many ready‑made tools for Linux observability, such as:

bcc-tools – a package of common BCC utilities.

bpftrace – a high‑level language for writing BPF programs.

tcptop – real‑time TCP traffic monitoring.

execsnoop – monitors process execution.

filetop – monitors file system activity.

trace – traces function calls.

funccount – counts function invocations.

opensnoop – monitors file open operations.

pidstat – monitors process performance.

profile – analyzes CPU usage.

6. Further Reading

Key references include Brendan Gregg’s *BPF Performance Tools*, the official eBPF website (ebpf.io), Cilium’s BPF and XDP guide, kernel BPF documentation, and the Awesome eBPF GitHub collection.