How Netcap Uses eBPF to Trace Kernel Network Packets and Solve Drop Issues

Background

During Linux kernel network development, packet loss is a common challenge. Traditional tools like tcpdump are limited in efficiency and depth. Leveraging the rapid development of eBPF, ByteDance STE created the next‑generation kernel network capture tool netcap and released it on GitHub.

Unlike tcpdump , which only hooks fixed points before sending or receiving packets, netcap can trace almost the entire kernel network stack (functions that receive an skb argument). It uses tcpdump‑style filters with skb as context, allowing developers to follow a packet’s full journey and locate loss points much faster.

Usage Examples

Example 1 : Check whether ICMP packets to IP 10.227.0.45 reach the expected kernel function.

<code>netcap skb -f icmp_rcv@1 -i eth0 -e "host 10.227.0.45" -t "-nnv"</code>

Options:

-i specifies the device associated with the skb (use with caution if dev is unset).

-e provides a tcpdump filter expression.

-t selects tcpdump’s output format.

Example 2 : Locate where packets to TCP port 9000 are dropped.

<code>netcap skb -f tracepoint:skb:kfree_skb -e "tcp port 9000" -S</code>

-S prints the kernel stack for the call, revealing the drop location.

To create a drop rule:

<code>iptables -A INPUT -p tcp --dport 9000 -j DROP</code>

Then observe the loss with netcap.

Aggregated tracing across multiple points:

<code>netcap skb -f tracepoint:net:netif_receive_skb,ip_local_deliver@1,ip_local_deliver_finish@3,icmp_rcv@1 -e "host 10.227.0.72 and icmp" -i eth0 --gather --gather-output-color cyan</code>

Design and Implementation

Netcap hooks functions via kprobe or tracepoint, extracts skb and sock structures, and passes packet data to user‑space through BPF maps.

The tool translates tcpdump filter syntax (cBPF) into a C function using the Cloudflare cbpfc library, embeds this function in the eBPF program, and forwards matching packets to a user‑space tcpdump instance for display or pcap output.

When sending packets, netcap attempts to reconstruct incomplete headers by inferring data from the sock structure, though some fields (e.g., IP ID) may differ from the actual packet.

Additional Uses and Extensions

Netcap can aggregate timestamps from multiple trace points to analyze performance bottlenecks.

Users can provide custom filter and action functions. Example filter:

<code>static inline int xcap_user_filter(void *ctx, void *pkt, u16 trace_index) {
    return 1;
}</code>

Example action with a custom data structure:

<code>struct xcap_user_extend {
    int a; // format: 0x%x
    uint32_t b;
    int64_t c;
    uint8_t x1; // format: %c
    uint8_t x2; // format: 0x%x
    uint16_t x3; // format: 0x%x
};
static inline int xcap_user_action(void *ctx, void *pkt, u32 pkt_len, struct xcap_user_extend *user, u16 trace_index) {
    user->a = 0x12345678;
    user->b = 1000;
    user->c = 2002;
    user->x1 = 'M';
    user->x2 = 0x11;
    user->x3 = 0xabcd;
    return 1;
}</code>

Future Outlook

Netcap aims to support DPDK, unify multi‑kernel versions, and resolve output interleaving when both tcpdump and custom actions write to stdout. The STE team invites contributions to further improve the tool.