Master Linux Tracing with bpftrace: Install, Run, and Explore Built‑in Tools

Introduction

bpftrace is an advanced Linux tracing tool and language built on eBPF and BCC. It compiles scripts to BPF bytecode, uses kernel probes (kprobes), user‑space probes (uprobes), and tracepoints to collect runtime information, then visualizes it to help developers locate deep bugs, security issues, and performance bottlenecks. The language is inspired by awk, C, DTrace, and SystemTap.

Project repository:

https://github.com/iovisor/bpftrace

Installation

Officially requires Linux kernel 4.9 or newer. Package commands for common distributions are:

Ubuntu

# For Ubuntu 19.04+ 
sudo apt-get install -y bpftrace

# On Ubuntu 16.04+ you can also use the snap package
sudo snap install --devmode bpftrace
sudo snap connect bpftrace:system-trace

Fedora

# Fedora 28+ includes bpftrace in the official repo
sudo dnf install -y bpftrace

Gentoo

# Install from the official Gentoo repository
sudo emerge -av bpftrace

Other distributions

Links for Debian, openSUSE, CentOS, etc.:

https://tracker.debian.org/pkg/bpftrace
https://software.opensuse.org/package/bpftrace
https://github.com/fbs/el7-bpf-specs/blob/master/README.md

Docker

$ docker run -v $(pwd):/output quay.io/iovisor/bpftrace:master-vanilla_llvm_clang_glibc2.23 \
  /bin/bash -c "cp /usr/bin/bpftrace /output"
$ ./bpftrace -V
v0.9.4

Built‑in Example Tools

bpftrace ships with several ready‑made scripts that also serve as language examples.

bashreadline.bt

Prints every command entered in a bash session.

# ./bashreadline.bt
Attaching 2 probes...
Tracing bash commands... Hit Ctrl‑C to end.
TIME      PID    COMMAND
06:40:06  5526   df -h
06:40:09  5526   ls -l
06:40:18  5526   echo hello bpftrace
...

biolatency.bt

Generates an I/O latency histogram.

# ./biolatency.bt
Attaching 3 probes...
Tracing block device I/O... Hit Ctrl‑C to end.
^C

@usecs:
[256, 512)          2 |
[512, 1K)          10 |@
[1K, 2K)          426 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2K, 4K)          230 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@                |
...

bitesize.bt

Shows disk I/O size distribution per process.

# ./bitesize.bt
Attaching 3 probes...
Tracing block device I/O... Hit Ctrl‑C to end.
...
@[cleanup]:
[4K, 8K)            2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
...

cpuwalk.bt

Displays which CPUs are executing processes.

# ./cpuwalk.bt
Attaching 2 probes...
Sampling CPU at 99hz... Hit Ctrl‑C to end.
^C

@cpu:
[0, 1)            130 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@    |
[1, 2)            137 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  |
[2, 3)             99 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                |
...

tcpconnect.bt

Tracks active TCP connections (calls to connect()).

# ./tcpconnect.bt
TIME      PID      COMM      SADDR      SPORT  DADDR        DPORT
00:36:45 1798396  agent     127.0.0.1  5001   10.229.20.82 56114
00:36:45 1798396  curl      127.0.0.1 10255   10.229.20.82 56606
00:36:45 3949059  nginx     127.0.0.1  8000   127.0.0.1    37780