Kubernetes Network Models: Underlay, Overlay, CNI Plugins, and Virtual Networking Techniques

The article explores the networking model in Kubernetes and provides an analysis of various network architectures.

Underlay Network Model : An underlay network refers to the physical infrastructure—switches, routers, DWDM—that forms the base topology for packet transmission. It can be a layer‑2 network (e.g., Ethernet) or a layer‑3 network (e.g., the Internet). Layer‑2 technologies include vlan , while layer‑3 relies on routing protocols such as OSPF and BGP .

In Kubernetes, the host acts as a router; pods learn routing entries to achieve cross‑node communication. Typical underlay implementations are the flannel host-gw mode and the Calico BGP mode.

flannel host‑gw : Each node must reside on the same layer‑2 network and functions as a router. Cross‑node traffic is forwarded via routing tables, effectively treating the cluster as an underlay network .

Calico BGP : BGP (Border Gateway Protocol) is a decentralized routing protocol that maintains IP routing tables (prefixes) between Autonomous Systems ( AS ). Calico runs a BGP client ( Felix ) that exchanges routes with a router reflector ( RR ) and peers ( BGP Peer ). The Bird daemon acts as the BGP client, reducing the number of BGP sessions required inside an AS.

IPVLAN & MACVLAN : These are NIC‑virtualization technologies. IPVLAN allows multiple IP addresses on a single MAC address, while MACVLAN provides multiple MAC addresses on a single IP address. Both are considered overlay networking techniques that flatten the pod network to the node level, offering higher performance and lower latency.

In Kubernetes, IPVLAN is used by CNI plugins such as multus and danm . multus (an Intel open‑source CNI) combines traditional CNI with SR‑IOV support and enables IPVLAN/MACVLAN functionality.

Overlay Network Model : An overlay network builds a virtual logical network on top of the underlay using tunneling protocols. It encapsulates packets to transport them across the physical infrastructure without altering the underlying topology.

Common tunnel technologies include:

Generic Routing Encapsulation ( GRE ) – encapsulates IPv4/IPv6 packets at L3.

Virtual Extensible LAN ( VxLAN ) – encapsulates L2 Ethernet frames inside UDP packets (default port 4789 ), expanding VLAN ID space from 4096 to 16 million.

IP in IP ( IPIP ) – encapsulates an IP packet within another IP packet; requires the kernel module ipip.ko (load with modprobe ipip ).

Both flannel and calico implement VxLAN using Linux kernel support (available since kernel 3.7, recommended on 3.9+). Flannel creates a VxLAN device named flannel.1 per node, maintains a VNID, and maps MAC addresses for cross‑node traffic. Example forwarding database entry:

$ bridge fdb
26:5e:87:90:91:fc dev flannel.1 dst 10.0.0.3 self permanent

Weave fastdp : Weave also uses VxLAN encapsulation, referred to as fastdp , and relies on the Linux openvswitch datapath module . It adds encryption and works in kernel versions ≥ 3.12; on older kernels (e.g., CentOS 7) it operates in user space as sleeve mode .

References:

https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#host-gw

https://projectcalico.docs.tigera.io/networking/bgp

https://www.weave.works/docs/net/latest/concepts/router-encapsulation/

https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin

https://github.com/nokia/danm