Kubernetes Networking Explained with 16 Detailed Diagrams

Underlay Network Model

Definition

An underlay network is the physical infrastructure—switches, routers, DWDM links—that carries packets between hosts. It can operate at Layer 2 (e.g., Ethernet) or Layer 3 (e.g., the Internet). Layer 2 technologies include vlan; Layer 3 protocols include OSPF and BGP.

Kubernetes underlay implementations

In Kubernetes the underlay often treats each node as a router, allowing Pods to learn routes for cross‑node communication. Typical implementations are Flannel’s host‑gw mode and Calico’s BGP mode.

Flannel host‑gw

Each node must reside on the same Layer 2 network and act as a router; cross‑node traffic is forwarded via routing tables, effectively simulating an underlay network. The cluster CIDR should be at least /16 to ensure that node‑level routing does not create unreachable networks.

Calico BGP

Calico uses the decentralized Border Gateway Protocol. It runs multiple daemons: flanneld maintains routes for Flannel, bird provides a BGP client that obtains routes from Felix and distributes them to BGP peers. A router reflector ( RR) reduces the number of internal BGP connections, so each node only needs a single RR connection within an iBGP mesh.

IPVLAN and MACVLAN

Both are NIC‑virtualization techniques. IPVLAN allows multiple IP addresses on a single NIC while sharing one MAC address. MACVLAN allows multiple MAC addresses on a single NIC, each virtual interface optionally lacking an IP. Because they flatten Pod networking to the node level, they are classified as overlay‑style mechanisms that improve performance and latency.

Multus

Multus is an Intel open‑source CNI that extends the default CNI and provides an SR‑IOV plugin. A pod can have three interfaces: eth0 (default Flannel network), a VF instantiated from host NIC ens2f0 named south0, and a VF from ens2f1 named north0 bound to the vfio‑pci driver. The SR‑IOV plugin moves the VF into the pod’s network namespace and sets the interface state to UP.

DANM

DANM is a Nokia open‑source CNI that brings carrier‑grade networking to Kubernetes. It supports SR‑IOV/DPDK and IPVLAN similar to Multus.

Overlay Network Model

Definition

An overlay network builds a virtual logical network on top of the underlay using tunneling protocols. It encapsulates packets to transport them across the physical infrastructure without modifying the underlying topology.

Common tunnel technologies

Generic Routing Encapsulation (GRE) – encapsulates IPv4/IPv6 packets at L3.

VxLAN – encapsulates L2 Ethernet frames in UDP (default port 4789) and expands VLAN IDs from 4096 to 16 million VNIDs.

IPIP

IP‑in‑IP uses the kernel module ipip.ko. Load it with modprobe ipip and verify with lsmod | grep ipip. Public clouds such as Azure may block IPIP traffic.

VxLAN in Kubernetes

Linux kernel support for VxLAN was merged in kernel 3.7 (stable from 3.9/3.10). Flannel creates a VxLAN device (e.g., flannel.1) with a VNID, maintains MAC addresses, and forwards packets across nodes. The forwarding database can be inspected with:

$ bridge fdb

Weave VxLAN

Weave implements VxLAN as fastdp using the Linux openvswitch datapath module and encrypts traffic. It runs in kernel space on versions ≥ 3.12; on older kernels (e.g., CentOS 7) it operates in user‑space “sleeve mode”.

IPIP vs VxLAN

Both are tunnel protocols used by Kubernetes. IPIP encapsulates an IP packet inside another IP header, while VxLAN encapsulates an Ethernet frame inside a UDP packet. This difference is reflected in packet captures: VxLAN appears as UDP port 4789 (Flannel’s default is 8472), whereas IPIP appears as an IP‑in‑IP header.

References

https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#host-gw

https://projectcalico.docs.tigera.io/networking/bgp

https://www.weave.works/docs/net/latest/concepts/router-encapsulation/

https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin

https://github.com/nokia/danm