Best Practices for Mixing Overlay and Underlay Networks with Kube‑OVN

In a recent technical sharing, Kube‑OVN author Liu Mengxin and a network expert discussed best practices for deploying Kube‑OVN in hybrid network scenarios, covering the features of Overlay and Underlay networks and how Kube‑OVN addresses their challenges.

Overlay Network

Overlay networking creates a tunneled network in Kubernetes, giving containers independent IP addresses and providing isolation and flexibility. Popular CNI solutions such as Flannel, Calico and Cilium support Overlay.

Low network dependency: tunnel encapsulation isolates container networking from the host.

Flexible configuration: gateways, DNS, etc., can be freely set.

Extensible functionality: software‑defined network can be expanded.

Security isolation: container IPs are not directly exposed.

Drawbacks:

Additional performance overhead due to encapsulation and decapsulation.

Increased operational complexity with two‑layer networking.

Interoperability issues with the underlying physical network requiring extra configuration.

Underlay Network

Underlay networking directly bridges the physical NIC, mapping container IP/MAC to the physical network, eliminating encapsulation and offering a simpler, more manageable solution.

High performance: no encapsulation overhead.

Native interoperability with the physical network.

Operationally similar to traditional networks, easy for network admins.

Drawbacks:

Strong dependency on the underlying network, leading to higher management cost and large address space consumption.

Complex changes: network modifications require support from the physical layer.

Limited extensibility due to physical constraints.

Hybrid deployment challenges when mixing Overlay and Underlay.

In Kubernetes, many workloads benefit from a hybrid approach: Overlay for isolated internal services and Underlay for high‑performance, externally‑exposed applications such as databases and middleware.

Hybrid Network Issues

Multus is an open‑source CNI plugin that enables Pods to have multiple network interfaces and IP addresses.

Multus Modes

Multi‑NIC mode: Pods can attach multiple CNI plugins, providing separate interfaces for Overlay and Underlay networks.

Single‑NIC mode: Pods use a single interface but can select different CNI plugins for different traffic.

Multi‑NIC advantages: satisfies both network demands, enables direct communication between Overlay and Underlay, and supports full network access.

Multi‑NIC drawbacks: address resource waste, service‑discovery and security policy limitations, added monitoring complexity, and application‑level awareness requirements.

Single‑NIC advantages: no application awareness needed, compatible with native Service and NetworkPolicy, saves IP resources.

Single‑NIC drawbacks: inter‑network connectivity issues and increased management complexity due to multiple plugins.

Related auxiliary projects such as multi‑networkpolicy API , multi‑networkpolicy‑iptables, multi‑networkpolicy‑tc, and multi‑service provide additional support for multi‑NIC environments.

Kube‑OVN Overview

Kube‑OVN, open‑sourced by Lingque Cloud in 2019, builds on Open vSwitch (OVS) and integrates Open Virtual Network (OVN) logical routers and switches to create a flexible SDN that supports both Overlay and Underlay networks.

Kube‑OVN Solution

The solution achieves Overlay‑Underlay inter‑connectivity by:

Connecting Overlay networks to logical routers via logical switches.

Bridging Underlay networks to physical switches through logical ports.

Creating logical ports in Underlay and linking them to Overlay routers, with routing rules to separate traffic.

Setting default gateways in Underlay Pods to point to the logical router ports.

Simplifying configuration by avoiding NAT and extra routing.

Advantages of the Kube‑OVN approach include transparent application experience, compatibility with native Service and NetworkPolicy, IP resource savings, and seamless Overlay‑Underlay communication without additional NAT or routing configuration.

- END -