Understanding Kubernetes Networking: Single‑Node Basics to Cross‑Host Overlay & Underlay

Kubernetes Network

Tags: container network, Kubernetes network

Mainly notes from the deep‑analysis Kubernetes series.

1. Single‑Node Container Network

Terminology

• Network stack : includes Network Interface, Loopback Device, Routing Table and Iptables rules, forming the basic environment for a process to send and receive network traffic.

• Bridge : a virtual network device that can be configured with IP and MAC addresses; the bridge acts as a virtual switch operating at the data‑link layer.

• Veth Pair : a virtual cable connecting a container to the bridge. It appears as two virtual NICs ( veth peer) that forward packets between the container’s network namespace and the bridge, even when the peers are in different network namespaces.

• ARP : protocol that maps an IP address to a MAC address at layer‑2.

• CAM table : the bridge’s MAC‑to‑port mapping table learned from traffic.

Host Network

Containers can use the host’s network namespace by starting with -net=host (e.g., docker run -d -net=host --name nginx-1 nginx). This gives better performance because the container shares the host’s network stack, but it introduces resource‑sharing issues such as port conflicts. Therefore, most scenarios prefer an isolated network namespace with its own IP and ports.

How Communication Works

The diagram shows the communication flow between two containers (C1 → C2) on a single host. The following steps illustrate the process:

# Create two containers for testing and install net‑tools
$ docker run -d -it --name c1 centos /bin/bash
$ docker exec -it c1 bash
$ yum install -y net-tools
$ docker run -d -it --name c2 centos /bin/bash
$ docker exec -it c2 bash
$ yum install -y net-tools

Both containers have a default route sending all traffic through eth0. Inside each container you can view the IP and routing table:

# Inside c1
$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.7  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ac:11:00:07
$ route
default         _gateway        0.0.0.0         UG    0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0

Similarly for c2 you see IP 172.17.0.8 and the same default route. The containers each have an ARP cache that already contains the MAC address of the bridge’s gateway ( _gateway).

When C1 wants to reach C2, it first checks its ARP cache for the MAC of C2’s IP (172.17.0.8). If missing, it sends an ARP request, which the bridge broadcasts to all attached interfaces. C2 replies with its MAC, the entry is cached, and the packet is forwarded via the bridge (acting as a virtual switch) to C2’s veth peer.

# C1 initiates ARP lookup
$ arp
Address                  HWtype  HWaddress           Flags Mask            Iface
_gateway                 ether   02:42:2e:8d:21:d6   C     0.0.0.0          eth0
# Ping triggers ARP request
$ ping 172.17.0.8
# Verify ARP cache now contains C2’s MAC
$ arp
Address                  HWtype  HWaddress           Flags Mask            Iface
172.17.0.8               ether   02:42:ac:11:00:08   C     0.0.0.0          eth0
_gateway                 ether   02:42:2e:8d:21:d6   C     0.0.0.0          eth0

2. Cross‑Host Container Communication

Cross‑host container communication is divided into Overlay and Underlay networks. Overlay only requires host‑to‑host IP reachability, while Underlay imposes stricter requirements on the underlying infrastructure (e.g., Flannel host‑gw needs the hosts to be on the same L2 domain).

Terminology

• Overlay Network : builds a virtual network on top of the existing host network, connecting all containers regardless of physical topology.

• Tun device : a layer‑3 virtual network interface in Linux that passes IP packets between kernel and user space.

• VXLAN : Virtual Extensible LAN, a kernel‑supported network‑virtualization technology that encapsulates L2 frames over an L3 network.

• VTEP : VXLAN Tunnel Endpoint, a device with both IP and MAC addresses that terminates VXLAN tunnels.

• BGP : Border Gateway Protocol, a native Linux routing protocol for large data‑center networks.

Overlay Communication

Overlay networking can be implemented in several ways. The article first examines Flannel’s original UDP mode.

Flannel UDP is the simplest early solution for cross‑host container networking, but it has the worst performance and is now deprecated. It remains useful for understanding basic overlay concepts.

Example scenario: four containers across two hosts, with Container‑1 requesting Container‑4.

Container‑1’s traffic leaves its network namespace via a veth pair to the host’s docker0 bridge, then follows the host’s routing table. Because the destination IP (100.96.2.2) is outside the docker0 subnet, the packet matches the route to the flannel0 TUN device:

# Node1 routing table
$ ip route
default via 10.168.0.1 dev eth0
100.96.0.0/16 dev flannel0 proto kernel scope link src 100.96.1.0
100.96.1.0/24 dev docker0 proto kernel scope link src 100.96.1.1
10.168.0.0/24 dev eth0 proto kernel scope link src 10.168.0.2

The flanneld daemon sees the destination 100.96.2.2, encapsulates the IP packet in a UDP datagram, and sends it to the corresponding flanneld on Node2. Node2’s daemon decapsulates the packet and forwards it to its local docker0 bridge, which finally delivers it to Container‑4 via the appropriate veth pair.

Flannel UDP’s performance drawback stems from the use of a TUN device, which forces three user‑kernel copies for each packet.

Calico and VXLAN

Calico implements an underlay BGP‑based approach. Its components are:

Calico CNI plugin : integrates Calico with Kubernetes.

BIRD : a BGP client that distributes routing information inside the cluster.

Felix : a daemon that programs Linux kernel routing tables (FIB) and creates the necessary network devices.

Calico does not create a virtual bridge; instead, each container’s veth pair connects directly to the host’s network namespace. Nodes exchange BGP routes so that each knows how to reach the others. Example routing tables:

# On node 192.168.0.2
$ ip route
10.20.0.2 dev cali1 scope link
10.20.0.3 dev cali2 scope link

# On node 192.168.0.3
$ ip route
10.20.1.2 dev cali3 scope link
10.20.1.3 dev cali4 scope link

Node‑to‑Node BGP scales quadratically (N²) and is practical for clusters with fewer than ~100 nodes. Larger clusters use a Route Reflector to centralise route distribution.

When hosts are not on the same L2 segment, Calico can fall back to an IP‑in‑IP overlay mode (Calico ipip).

Underlay Mode – BGP Example

In a typical BGP setup, two border routers (Route1, Route2) inject LAN routes into the global routing table, enabling full L3 connectivity across the data center.

Flannel host‑gw Mode

Flannel host‑gw uses a three‑layer switch (CNI0) that behaves like a L2 switch while also providing IP routing. The Flannel daemon runs as a DaemonSet, launching a flanneld process on each node to maintain routing entries such as:

# Example route added by flanneld
192.168.1.0/24 via 10.20.0.3 dev eth0

When an IP packet is sent, the kernel uses the next‑hop MAC address from the ARP cache, so host‑to‑host L2 connectivity is required.

Summary of Key Points

Container networking on a single host relies on the Linux bridge, veth pairs, and ARP for L2 communication.

Host networking ( -net=host) offers performance but sacrifices isolation.

Cross‑host communication can be achieved with Overlay networks (Flannel UDP, Calico ipip, VXLAN) or Underlay networks (BGP‑based Calico, Flannel host‑gw).

Overlay solutions encapsulate traffic (UDP, IP‑in‑IP, VXLAN) and may incur extra copies; Underlay solutions depend on underlying L2/L3 topology.

Calico’s BGP mode scales poorly without a Route Reflector; Flannel host‑gw requires hosts to be on the same L2 segment.