Understanding Kubernetes Underlay and Overlay Network Models: A Deep Dive

Overview

This article explores the network models in Kubernetes and analyzes various models.

Underlay Network Model

What is an Underlay Network

The underlay network refers to the physical network infrastructure such as switches, routers, and DWDM that link devices into a physical topology responsible for packet transmission.

The underlay network can be layer‑2 (e.g., Ethernet) or layer‑3 (e.g., Internet). Layer‑2 technologies include VLAN, while layer‑3 uses protocols such as OSPF and BGP.

Underlay network in Kubernetes

In Kubernetes, a typical underlay example is using the host as a router, with Pods learning routes to achieve cross‑node communication.

Typical implementations are Flannel host‑gw mode and Calico BGP mode.

Flannel host‑gw

In flannel host‑gw mode each node must be on the same layer‑2 network and acts as a router, forming an underlay network.

Notes: Because routing is used, the cluster CIDR must be at least /16 to ensure node and pod networks are separate.

Calico BGP

BGP (Border Gateway Protocol) is a decentralized routing protocol that maintains IP routing tables or prefix tables to provide connectivity between autonomous systems.

Unlike flannel, Calico provides a BGP solution. Flannel uses the flanneld process to maintain routes, while Calico consists of multiple daemons; the Brid process acts as a BGP client and router reflector, and the BGP client obtains routes from Felix and distributes them to BGP peers. The router reflector reduces the number of internal BGP connections, and Bird works as the BGP client.

IPVLAN & MACVLAN

IPVLAN and MACVLAN are NIC virtualization technologies. IPVLAN allows multiple IP addresses on a single physical NIC with a shared MAC, while MACVLAN allows multiple MAC addresses on a single NIC, each virtual interface may lack an IP.

These are overlay network technologies that flatten the pod network to the node level, providing higher performance and lower latency.

Virtual bridge: creates a veth pair, one end in the container, the other in the host root namespace, allowing packets to pass through the bridge.

Multiplexing: uses an intermediate device exposing multiple virtual NICs, distinguished by MAC/IP.

Hardware offload (SR‑IOV): assigns a virtual NIC to each pod, using VF interfaces that share a physical PCIe channel.

In Kubernetes, IPVLAN is used by CNI plugins such as multus and danm.

multus

Multus is an Intel open‑source CNI that combines traditional CNI with multus and provides an SR‑IOV plugin, enabling pods to use IPVLAN/MACVLAN.

When a new pod is created, the SR‑IOV plugin moves the VF into the pod’s network namespace, sets the interface name from the CNI config, and brings the VF up.

Example pod with three interfaces: eth0 (flannel default), VF on host port ens2f0 (south0), and VF on host port ens2f1 (north0) using vfio-pci driver.

danm

DANM is Nokia’s open‑source CNI project that brings carrier‑grade networking to Kubernetes, supporting SR‑IOV, DPDK, and IPVLAN.

Overlay Network Model

What is an Overlay

An overlay network uses virtualization techniques to build a logical network on top of an underlay network without changing the physical infrastructure. It typically employs tunnel protocols to encapsulate packets.

Common tunnel technologies

Generic Routing Encapsulation (GRE) encapsulates IPv4/IPv6 packets at L3.

VxLAN (Virtual Extensible LAN) encapsulates L2 Ethernet frames into UDP packets, using port 4789 by default.

Typical overlay implementations in Kubernetes include flannel and calico using VxLAN or IPIP modes.

IPIP

IP in IP is a tunnel protocol implemented via the Linux kernel module ipip.ko; it can be loaded with modprobe ipip and verified with lsmod | grep ipip.

In Kubernetes, IPIP works similarly to VxLAN, encapsulating packets within IP packets.

Notes: Public clouds may block IPIP traffic, e.g., Azure.

VxLAN

Both flannel and calico implement VxLAN using Linux kernel support introduced in kernel 3.7.0; newer kernels (≥3.9.0) are recommended.

In Kubernetes, flannel maintains a VxLAN device (flannel.1) per node, assigning a VNID and routing MAC addresses to forward traffic across nodes.

To view the forwarding database:

$ bridge fdb
26:5e:87:90:91:fc dev flannel.1 dst 10.0.0.3 self permanent

Notes: VxLAN uses port 4789; flannel defaults to UDP port 8472, so captures may show UDP.

Tunnels are abstract concepts that encapsulate packets for transport over the physical network and then decapsulate them.

weave VxLAN

Weave also uses VxLAN, referred to as fastdp (fast data path), built on the openvswitch datapath module and encrypts traffic.

Notes: fastdp works on Linux kernel ≥3.12; older kernels use user‑space sleeve mode.

Reference

https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#host-gw https://projectcalico.docs.tigera.io/networking/bgp https://www.weave.works/docs/net/latest/concepts/router-encapsulation/ https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin https://github.com/nokia/danm