Mastering pwru: A Step‑by‑Step Guide to eBPF Packet Tracing with Cilium

What is pwru?

pwru (packet, where are you?) is an eBPF‑based network packet tracing utility released by Cilium. It offers fine‑grained visibility into packet processing inside the Linux kernel, helping operators quickly locate where packets are dropped or altered.

Installation Requirements

pwru requires a Linux kernel newer than 5.5 and, for full functionality, kernel version 5.9+ with the following configuration options enabled: --output-skb Typical kernel config flags include BPF, BTF, and related debugging options (see the accompanying screenshot).

Installing pwru

Verify kernel configuration: zgrep CONFIG_DEBUG_INFO_BTF /proc/config.gz Download the latest pre‑built static binary and extract it:

wget https://github.com/cilium/pwru/releases/latest/download/pwru-linux-amd64.tar.gz
 tar -xzf pwru-linux-amd64.tar.gz
 sudo ./pwru --help

Command‑Line Options

pwru supports many filters and output modifiers. Key flags include:

--filter-dst-ip string          filter destination IP address
--filter-dst-port uint16        filter destination port
--filter-func string           filter kernel functions (exact or RE2 regex)
--filter-mark uint32            filter skb mark
--filter-netns uint32           filter netns inode
--filter-proto string           filter L4 protocol (tcp, udp, icmp)
--filter-src-ip string          filter source IP address
--filter-src-port uint16        filter source port
--output-limit-lines uint       stop after a number of events
--output-meta                   print skb metadata
--output-relative-timestamp     print relative timestamps per skb
--output-skb                    print raw skb data
--output-stack                  print kernel stack trace
--output-tuple                  print L4 tuple (src/dst IP and ports)

Typical Use Cases

Below are common scenarios demonstrated with pwru.

1. Simple packet tracing

sudo ./pwru

2. Filter by destination host

sudo ./pwru 'host 1.1.1.1'

3. Filter by port and protocol (HTTP traffic)

sudo ./pwru 'tcp port 80'

4. Show L4 tuple for better identification

sudo ./pwru --output-tuple 'host 8.8.8.8'

5. Debug packet drops in the network stack

sudo ./pwru --output-stack --output-tuple 'host 10.0.0.1'

6. Analyze TCP connection issues with flags

sudo ./pwru --output-tcp-flags --output-tuple 'tcp port 443'

Case Study: Tracing iptables DROP

The following GIF illustrates how pwru quickly pinpoints a packet being dropped by an iptables rule. Before adding the rule, packets flow normally; after adding a DROP rule, pwru shows the packet reaching nf_hook_slow, being marked NF_DROP, and the kernel freeing the skb.

By examining the stack trace and metadata, operators can identify the exact hook and rule responsible for the drop, dramatically reducing debugging time.