nettrace: An eBPF‑Based Tool for Network Packet Tracing, Diagnosis and Drop Monitoring in Cloud‑Native Environments

nettrace is an eBPF‑powered command‑line tool that integrates network packet tracing, fault diagnosis and anomaly monitoring. It was created to address the growing complexity of Linux networking in cloud‑native scenarios, where packets may traverse NAT, GRE, IPVS and other middleboxes before reaching the destination.

The tool fills gaps left by traditional utilities such as tcpdump , ftrace , kprobe , BCC and dropwatch , which either lack kernel‑level visibility, require deep kernel knowledge, or provide only single‑purpose functionality.

Key capabilities :

Network packet tracing – records the full lifecycle of a packet as it moves through the kernel stack.

Network fault diagnosis – matches observed events against a built‑in knowledge base to suggest root causes and remediation steps.

Network anomaly monitoring – continuously runs in production to detect and report abnormal traffic.

Droptrace replacement – nettrace --drop provides drop‑reason monitoring similar to the legacy droptrace tool.

Installation

nettrace is written in C and built on libbpf. It can be installed via a pre‑compiled RPM or binary. The tool has been added to the OpenCloudOS software repository, so on a system that inherits yum you can install it with:

sudo yum install nettrace

It requires a Linux kernel version 4.14 or newer.

Command‑line interface

Running nettrace -h prints a comprehensive list of options:

$ nettrace -h
nettrace: a tool to trace skb in kernel and diagnose network problem
Usage:
    -s, --saddr      filter source ip address
    --saddr6         filter source ip v6 address
    -d, --daddr      filter dest ip address
    --daddr6         filter dest ip v6 address
    --addr           filter source or dest ip address
    --addr6          filter source or dest ip v6 address
    -S, --sport      filter source TCP/UDP port
    -D, --dport      filter dest TCP/UDP port
    -P, --port       filter source or dest TCP/UDP port
    -p, --proto      filter L3/L4 protocol, such as 'tcp', 'arp'
    --pid            filter by current process id(pid)
    -t, --trace      enable trace group or trace
    --ret            show function return value
    --detail         show extern packet info, such as pid, ifname, etc
    --date           print timestamp in date-time format
    --basic          use 'basic' trace mode, don't trace skb's life
    --diag           enable 'diagnose' mode
    --diag-quiet     only print abnormal packet
    --diag-keep      don't quit when abnormal packet found
    --hooks          print netfilter hooks if dropping by netfilter
    --drop           skb drop monitor mode, for replace of 'droptrace'
    --drop-stack     print the kernel function call stack of kfree_skb
    -v               show log information
    --debug          show debug information
    -h, --help       show help information

Example: tracing an ICMP packet (lifecycle mode)

sudo ./nettrace -p icmp
begin trace...
***************** ffff889be8fbd500,ffff889be8fbcd00 ***************
[1272349.614564] [dev_gro_receive] ICMP: 169.254.128.15 -> 172.27.0.6 ping request, seq: 48220
[1272349.614579] [__netif_receive_skb_core] ICMP: 169.254.128.15 -> 172.27.0.6 ping request, seq: 48220
... (truncated for brevity) ...
[1272349.614711] [consume_skb] ICMP: 169.254.128.15 -> 172.27.0.6 ping request, seq: 48220

The output shows each function the packet passes through, allowing users familiar with the kernel network stack to pinpoint where a problem occurs.

Diagnostic mode

Adding the --diag flag enables knowledge‑based analysis. The tool annotates the trace with INFO, WARN and ERROR levels and suggests fixes. Example output:

./nettrace -p icmp --diag --saddr 192.168.122.8
begin trace...
***************** ffff889fad356200 ***************
[3445.575957] [__netif_receive_skb_core] ICMP: 192.168.122.8 -> 10.123.119.98 ping request, seq: 0
[3445.575978] [nf_hook_slow] ICMP: 192.168.122.8 -> 10.123.119.98 ping request, seq: 0 *ipv4 in chain: PRE_ROUTING*
[3445.575990] [nft_do_chain] ICMP: 192.168.122.8 -> 10.123.119.98 ping request, seq: 0 *iptables table:nat, chain:PREROUT* *packet is accepted*
... (truncated) ...
---------------- ANALYSIS RESULT ---------------------
[1] WARNING happens in nf_nat_manip_pkt(netfilter):
        NAT happens (packet address will change)
    fix advice:
        check your NAT configuration
[2] ERROR happens in nf_hook_slow(netfilter):
        packet is dropped by netfilter (NF_DROP)
    fix advice:
        check your netfilter rule

This mode is especially useful for users without deep kernel expertise.

Netfilter and hook support

nettrace can display the iptables tables and chains a packet traverses, and when a drop occurs it reports the responsible hook functions. Example:

... (output omitted) ...
[1] ERROR happens in nf_hook_slow(netfilter):
        packet is dropped by netfilter (NF_DROP)
    following hook functions are blamed:
        nft_do_chain_ipv4
    fix advice:
        check your netfilter rule

Drop monitoring

Running nettrace --drop monitors all skb drop events and, on kernels that support the skb drop reason feature, prints the reason code. Example:

nettrace --drop
begin trace...
[142.097193] TCP: 162.241.189.135:57022 -> 172.27.0.6:22 seq:299038593, ack:3843597961, flags:AR, reason: NOT_SPECIFIED, tcp_v4_rcv+0x81
[146.136576] TCP: 127.0.0.1:43582 -> 127.0.0.1:9999 seq:3819454691, ack:0, flags:S, reason: NO_SOCKET, tcp_v4_rcv+0x81

When the kernel lacks this feature, the reason column is omitted.

Other scenarios

TCP port not listened – nettrace reports a WARNING and suggests checking the target port.

XDP‑program drop – nettrace identifies the drop source as an XDP eBPF program and points to the relevant hook.

Overall, nettrace provides a unified, easy‑to‑use interface for deep network troubleshooting, diagnosis and monitoring in modern cloud‑native Linux environments.