Linux Kernel Networking: How veth, Bridges, and Overlay Forwarding Power Container Communication

veth – virtual Ethernet pair

veth is a pair of virtual network interfaces created by the Linux kernel. One end (named eth0) resides inside a container’s network namespace, the other end (named veth<id>) stays in the host namespace. The pair forms a virtual cable that carries packets between the container and the host.

Creation and configuration

# Kernel‑level equivalent of Docker’s actions
ip link add vethabc type veth peer name eth0

# Attach the host‑side device to the default bridge docker0
ip link set vethabc master docker0
ip link set vethabc up

# Configure the container‑side device inside the container
ip addr add 172.17.0.2/16 dev eth0
ip link set eth0 up
ip route add default via 172.17.0.1

When a container runs ping 8.8.8.8, the ICMP request leaves eth0, traverses the veth pair to docker0, and is forwarded to the physical NIC for external delivery. The reply follows the reverse path.

Demo and packet capture

Start two busybox containers:

docker run -itd --name container1 busybox
docker run -itd --name container2 busybox

Inspect their IP addresses (e.g., 172.17.0.2 and 172.17.0.3 ) and ping between them: docker exec -it container1 ping 172.17.0.3 Capture traffic on all veth interfaces and analyse with Wireshark:

sudo tcpdump -i veth+ -w veth.pcap

Linux bridge – layer‑2 switch

A Linux bridge operates at the data‑link layer. It learns source MAC addresses from incoming frames and forwards frames based on its MAC table. If the destination MAC is known, the frame is sent only to the corresponding port; otherwise it is flooded to all ports except the source. Frames whose source and destination map to the same port are discarded, reducing broadcast traffic.

The default Docker bridge docker0 can be listed with:

ip link show
brctl show docker0

When multiple containers are attached, each container’s host‑side veth is linked to docker0. The bridge learns each container’s MAC address, enabling direct forwarding without broadcasting.

Bridge configuration and performance tuning

Assign a custom subnet (e.g., 192.168.1.1/24 ) by editing /etc/docker/daemon.json and restarting Docker:

{
  "bip": "192.168.1.1/24"
}

Manually add or remove a veth from the bridge:

sudo ip link set veth123 master docker0   # add
sudo ip link set veth123 nomaster        # remove

Adjust MTU to avoid fragmentation in overlay scenarios: sudo ip link set docker0 mtu 1450 Enable promiscuous mode for full traffic capture on a specific veth: sudo ip link set veth123 promisc on Shorten MAC‑table aging time to keep the table fresh under high load.

Overlay networks – virtual networks over the physical underlay

Overlay networks create a virtual layer on top of the physical network, allowing containers on different hosts to appear as if they share a common L2 segment. Packets are encapsulated, transmitted over the underlay (e.g., Ethernet), and then decapsulated at the destination.

Encapsulation process

The original Ethernet frame is wrapped in an outer header that carries a virtual network identifier (VNI) and the tunnel’s IP/UDP information.

The underlay forwards the encapsulated packet using standard IP routing.

The remote host’s VTEP (Virtual Tunnel End Point) removes the outer headers, restoring the original frame for delivery to the target container.

Common overlay protocols

VxLAN – MAC‑in‑UDP encapsulation, supports up to 16 million VNIs.

# Create a VxLAN interface with VNI 100, remote VTEP 10.0.0.2, using eth0 as the underlay
sudo ip link add vxlan100 type vxlan id 100 remote 10.0.0.2 dev eth0
sudo ip addr add 192.168.100.1/24 dev vxlan100
sudo ip link set vxlan100 up
# Attach to a bridge for container attachment
sudo ip link set vxlan100 master br‑overlay

GRE – Generic Routing Encapsulation, simpler but less scalable than VxLAN.

# Create a GRE tunnel
sudo ip tunnel add gre‑tun mode gre local 203.0.113.1 remote 203.0.113.2 ttl 64
sudo ip addr add 10.10.0.1/24 dev gre‑tun
sudo ip link set gre‑tun up
# Route remote subnet through the tunnel
sudo ip route add 10.10.1.0/24 dev gre‑tun

Overlay in a Kubernetes cluster

When a CNI plugin that supports VxLAN (e.g., Flannel, Calico) is used, each node creates a VTEP device. Example for node 1:

ip link add vxlan100 type vxlan id 100 remote 192.168.1.102 dev eth0
ip addr add 10.244.0.1/24 dev vxlan100
ip link set vxlan100 up
ip link set vxlan100 master cni0

Pods receive IPs from the overlay subnet (e.g., 10.244.0.2 and 10.244.1.3). When a pod on node 1 sends traffic to a pod on node 2, the packet travels from the pod’s eth0 through its veth pair, reaches the node’s bridge, is handed to the VTEP, encapsulated in VxLAN, traverses the physical network, and is decapsulated on node 2 before being delivered to the destination pod.

Verification commands:

# Show VxLAN device details
ip -d link show vxlan100
# Capture VxLAN traffic (default UDP port 8472)
sudo tcpdump -i eth0 udp port 8472 -vv

Coordinated operation of veth, bridge, and overlay

A packet generated in container C1 follows this layered path:

Leaves eth0 inside C1 and passes through the container‑side veth.

The host‑side veth delivers the frame to the node’s bridge (e.g., cni0).

If the bridge does not have a local MAC entry for the destination, it forwards the frame to the node’s VTEP.

The VTEP encapsulates the frame in VxLAN, adds outer IP/UDP headers, and sends it over the underlay.

At the destination node, the VTEP decapsulates the packet, hands it to the bridge, which forwards it to the target container’s veth and finally to eth0 of container C2.

Successive tcpdump captures on the veth, bridge, and physical interfaces can illustrate each stage.

Problem diagnosis and optimization

Use ping and traceroute inside containers to locate latency sources (veth, bridge, or overlay).

Adjust VxLAN MTU to avoid fragmentation, e.g.: sudo ip link set vxlan100 mtu 1450 Inspect bridge MAC tables with brctl macshow and reduce aging time to clear stale entries.

Verify VTEP configuration consistency (VNI, remote IP) with: ip -d link show vxlan* Ensure firewall rules allow VxLAN UDP traffic (default ports 4789 or 8472):

sudo iptables -A INPUT -p udp --dport 4789 -j ACCEPT

When containers cannot communicate, inspect in order: container IP/routing, host‑side veth status, bridge bindings, and overlay tunnel health.

By walking through each layer—from the container’s network namespace, through the veth pair, the Linux bridge, and finally the overlay tunnel—operators can pinpoint and resolve connectivity issues efficiently.