Understanding Kubernetes Network Models: Underlay, Overlay, and CNI Plugins

Overview

This article explores the networking model in Kubernetes and analyzes various network models.

Underlay Network Model

What is an Underlay Network

An underlay network refers to the physical infrastructure of network devices—switches, routers, DWDM—that connect via physical media to form a topology responsible for packet transport.

The underlay can be layer‑2 (e.g., Ethernet) or layer‑3 (e.g., the Internet). Layer‑2 typically uses VLANs, while layer‑3 relies on routing protocols such as OSPF and BGP.

Underlay in Kubernetes

In Kubernetes, a common underlay example is using the host as a router, with Pods learning routing entries to achieve cross‑node communication. Typical implementations include Flannel’s host‑gw mode and Calico’s BGP mode.

Flannel host‑gw

In host‑gw mode each node must reside on the same layer‑2 network and act as a router; cross‑node traffic is handled via routing tables, effectively treating the network as an underlay.

Note: Because routing is used, the cluster CIDR should be at least /16 to ensure that node‑level routes and pod‑level routes do not conflict.

Calico BGP

BGP (Border Gateway Protocol) is a decentralized autonomous routing protocol that maintains IP routing tables between autonomous systems.

Calico provides a BGP‑based solution similar to Flannel’s host‑gw in network model, but its architecture uses multiple daemons: a BGP client (Brid) obtains routes from Felix and distributes them to peers, while a Route Reflector (RR) reduces the number of internal BGP sessions. In an iBGP setup, the BGP client connects to a single RR, often a real router, with Bird acting as the BGP client.

IPVLAN & MACVLAN

IPVLAN and MACVLAN are NIC virtualization technologies. IPVLAN allows a single physical NIC to have multiple IP addresses sharing one MAC, while MACVLAN gives one NIC multiple MAC addresses, each with its own virtual interface.

These are overlay‑type solutions that flatten the Pod network to the same level as the node network, offering higher performance and lower latency.

Virtual bridge: creates a veth pair, one end in the container, the other in the host root namespace, allowing packets to pass directly between container and host network stacks.

Multiplexing: a middle network device exposes multiple virtual interfaces; containers are distinguished by MAC/IP addresses.

Hardware switch: each Pod receives a virtual NIC (e.g., via SR‑IOV), providing near‑bare‑metal connectivity.

In Kubernetes, typical CNI plugins for IPVLAN are Multus and DANM.

Multus

Multus is an Intel‑open‑source CNI that combines traditional CNI with additional plugins and provides an SR‑IOV CNI to connect Pods to SR‑IOV virtual functions, leveraging IPVLAN/MACVLAN capabilities.

When a new Pod is created, the SR‑IOV plugin moves the VF into the Pod’s network namespace, sets the interface name from the CNI configuration, and brings the VF up.

Below is a diagram of a Pod with three interfaces (eth0, VF‑south0, VF‑north0) using Multus and the SR‑IOV CNI.

eth0 – default Flannel network.

VF – instance of the host physical port ens2f0 (Intel X710‑DA4), named south0 inside the Pod.

VF – instance of host port ens2f1, using the DPDK driver, named north0 inside the Pod.

Terms: NIC – network interface card. SR‑IOV – single root I/O virtualization, allowing multiple VFs to share a PCIe device. VF – virtual function, derived from a physical function (PF). PF – PCIe physical function with full control over the device. DPDK – Data Plane Development Kit.

Alternatively, a host interface can be moved directly into the Pod’s network namespace, placing the Pod network on the same plane as the node network.

DANM

DANM is a Nokia‑open‑source CNI project that brings carrier‑grade networking to Kubernetes, supporting SR‑IOV/DPDK and IPVLAN similar to Multus.

Overlay Network Model

What is an Overlay

An overlay network uses virtualization techniques to build a logical network on top of an underlay without changing the physical infrastructure. It typically employs tunneling protocols to encapsulate packets for transport.

Common Tunnel Technologies

Generic Routing Encapsulation (GRE) – encapsulates IPv4/IPv6 packets at L3.

VxLAN (Virtual Extensible LAN) – encapsulates L2 Ethernet frames inside UDP (default port 4789), extending VLAN IDs from 4096 to 16 million.

Typical overlay implementations in Kubernetes include Flannel and Calico’s VxLAN and IPIP modes.

IPIP

IP‑in‑IP is a tunnel protocol that encapsulates an IP packet within another IP packet, requiring the kernel module ipip.ko. Load it with modprobe ipip if not present.

In Kubernetes, IPIP works similarly to VxLAN but encapsulates the packet within its own IP header rather than UDP.

Note: Public clouds may block IPIP traffic (e.g., Azure).

VxLAN

Both Flannel and Calico implement VxLAN using Linux kernel support, which was merged in kernel 3.7.0; stable operation is recommended on kernels 3.9.0 or newer.

In a Kubernetes VxLAN setup (e.g., Flannel), each node runs a VxLAN device (flannel.1) identified by a VNID. When cross‑node traffic occurs, the local VxLAN device learns the remote VxLAN MAC address and forwards encapsulated packets accordingly.

Example of viewing the forwarding database:

$ bridge fdb
26:5e:87:90:91:fc dev flannel.1 dst 10.0.0.3 self permanent

Note: VxLAN uses UDP port 4789; Flannel’s default UDP port is 8472, so packet captures will show UDP traffic.

Weave VxLAN

Weave also uses VxLAN (called fastdp) but relies on the Open vSwitch datapath module and encrypts traffic.

Note: fastdp requires Linux kernel 3.12+; on older kernels (e.g., CentOS 7) it runs in user space as “sleeve mode”.

Source: https://www.sohu.com/a/612068912_121124373