Understanding Underlay and Overlay Network Models in Kubernetes

Overview

This article explores the network models used in Kubernetes, analyzing both underlay and overlay approaches.

Underlay Network Model

What is Underlay Network

The Underlay Network refers to the physical infrastructure of network devices such as switches, routers, and DWDM, forming the physical topology that carries packets between networks.

The underlay can be either Layer‑2 (e.g., Ethernet) or Layer‑3 (e.g., Internet). Layer‑2 technologies include vlan , while Layer‑3 uses routing protocols like OSPF and BGP .

Underlay Network in Kubernetes

In Kubernetes, a typical underlay example treats each host as a router, allowing Pods to learn routes for cross‑node communication.

Common underlay‑based CNI modes are flannel host‑gw and calico BGP .

flannel host‑gw

In the flannel host‑gw mode, every node must reside on the same Layer‑2 network and acts as a router, effectively turning the underlay into the Pod network.

Notes: Because routing is used, the cluster CIDR should be at least /16 to ensure that cross‑node Nodes act as a single network and Pods as a separate network; otherwise routing tables may become unreachable.

Calico BGP

BGP (Border Gateway Protocol) is a decentralized routing protocol that maintains IP routing tables or prefix tables to enable connectivity between Autonomous Systems (AS).

Calico’s BGP implementation differs from flannel: Calico runs multiple daemons, with the Brid process acting as a BGP client that obtains routes from Felix and distributes them to BGP peers. A Route Reflector ( RR ) reduces the number of internal BGP connections.

IPVLAN & MACVLAN

IPVLAN allows a physical NIC to have multiple IP addresses while sharing a single MAC, whereas MACVLAN gives a NIC multiple MAC addresses, optionally without IPs. Both are NIC‑virtualization techniques and belong to the overlay network category, flattening Pod networks to the node level for higher performance.

multus

multus is an Intel‑open‑source CNI that combines traditional CNI with additional plugins, providing SR‑IOV support and enabling IPVLAN/MACVLAN functionality.

When a new Pod is created, the SR‑IOV plugin moves the VF into the Pod’s network namespace, sets the interface name from the CNI config, and brings the VF up.

Notes: The term “NIC” stands for network interface card; “SR‑IOV” is single root I/O virtualization; “VF” is Virtual Function; “PF” is Physical Function; “DPDK” is Data Plane Development Kit.

Overlay Network Model

What is Overlay

An overlay network builds a virtual logical network on top of the underlay using tunneling protocols, encapsulating packets to transmit across the physical infrastructure.

Common Tunnel Technologies

Generic Routing Encapsulation (GRE) – encapsulates IPv4/IPv6 packets at L3.

VxLAN (Virtual Extensible LAN) – encapsulates L2 Ethernet frames in UDP, using port 4789 and expanding VLAN IDs to 16 million VNIDs.

IPIP

IPIP (IP in IP) is another tunnel protocol implemented via the Linux kernel module ipip.ko . It can be loaded with modprobe ipip and verified with lsmod | grep ipip .

VxLAN

Both Flannel and Calico implement VxLAN using Linux kernel support (available since kernel 3.7, recommended on 3.9+). In Kubernetes, the flannel daemon creates a VxLAN device (e.g., flannel.1 ) with a VNID and maintains routing for cross‑node traffic.

$ bridge fdb
26:5e:87:90:91:fc dev flannel.1 dst 10.0.0.3 self permanent

Notes: VxLAN uses port 4789, but Flannel defaults to UDP port 8472; packet captures will show UDP packets.

weave VxLAN

Weave also uses VxLAN (called fastdp) and relies on the Open vSwitch datapath module. It encrypts traffic and, on kernels < 3.12, runs in user‑space (sleeve mode).

Reference

https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#host-gw https://projectcalico.docs.tigera.io/networking/bgp https://www.weave.works/docs/net/latest/concepts/router-encapsulation/ https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin https://github.com/nokia/danm