How ByteDance Uses eBPF netkit to Replace veth for Faster Container Networking

ByteDance engineers summarize that netkit, built on eBPF, provides a faster alternative to virtual Ethernet for container networking.

Chinese social‑media company ByteDance’s software engineers are early adopters of the Linux kernel’s newly released netkit feature, which offers a quicker way for containers to communicate within a cluster.

Netkit debuted in Linux kernel 6.7 (December 2023) as an eBPF‑programmable kernel network device, not to be confused with the discontinued Netkit for creating virtual networks on a single server. It is promoted as a simplification of container networking.

Like many cloud‑native companies, ByteDance traditionally uses virtual Ethernet (veth) to connect containers. veth, present in Linux since 2008, creates private container networks while still allowing external communication.

Problems with veth

Although veth seems suitable for container networking, practitioners quickly discovered several bottlenecks that reduce inter‑container communication speed. Each packet must traverse two network stacks—one for the sender and one for the receiver—even when both containers reside on the same host.

Virtual Ethernet forces packets through the L2 layer, requiring ARP routing to locate the destination address.

Why netkit is faster

Embedded in the kernel, netkit can intercept packets before they reach the network stack and internally route them when the destination is on the same host. In other words, everything is handled at layer 3, avoiding the extra L2 processing.

According to an eBPF Foundation case study, ByteDance observed a 10% performance improvement and reduced CPU usage after eliminating these “soft bottlenecks.” This is good news for a performance‑focused company.

ByteDance’s investigation

ByteDance, founded in 2012, operates platforms such as TikTok, Douyin, Toutiao, and others, serving nearly 2 billion users.

Partnering closely with Isovalent, ByteDance was the first to support netkit in Cilium version 1.16. However, netkit requires Linux kernel 6.7 or newer, while ByteDance currently runs kernel 5.15 due to operational constraints, so they are back‑porting eBPF (and the tcx traffic‑control extension) and updating their Container Network Interface (CNI).

Successful rollout

ByteDance has updated multiple kernels and CNI components with eBPF, deploying netkit in “dozens of clusters” with few reported issues.

Senior engineer Chen Tang said at last year’s eBPF summit, “We have not seen any incident reports about netkit. I can say with confidence that netkit is trustworthy.”

As with any production system, the upgrade strategy avoids downtime, so veth remains temporarily alongside netkit.

New containers will use netkit, while existing ones continue with veth. Engineers studied how the two protocols interact and how veth can serve as a fallback if netkit fails.

Netkit requires no extra configuration; once in the kernel, a netlink socket can create netkit devices. ByteDance added support in the golang‑netlink package and iproute2.

Setting up veth and netkit is similar, but for optimal performance the eBPF program should attach to the container side of the netkit pair.

The company also built a proxy to manage eBPF programs on hosts, handling tasks such as splitting ACL rules into BPF map entries and dumping kernel‑debug information.