Which Multi‑Cluster Networking Solution Is Right for Your Kubernetes Fleet?

Kubernetes clusters are increasingly deployed across multiple data centers, clouds, and edge locations, making cross‑cluster communication a key challenge. This article reviews six prominent solutions, explaining how they work, their benefits, and their drawbacks.

1. Underlay Network

Underlay networks rely on the underlying infrastructure to provide layer‑2 or layer‑3 connectivity between clusters, such as VPC peering or physical routing. When the physical network is connected, container networks across clusters communicate naturally.

Advantages

CNI‑centric and simple; no extra components required. Clear architecture that delegates cross‑cluster traffic to the underlying network.

Limitations

Depends on specific CNI implementations; not suitable for all environments. Hard to bridge heterogeneous clouds or on‑premise networks. Provides only basic container networking; higher‑level features like service discovery, DNS, or network policies are missing. Lacks fine‑grained control; all clusters are connected in a single blanket.

2. Overlay CNI Meshes

When an underlay is unavailable, some CNI plugins implement cross‑cluster connectivity at the overlay level. Notable examples include Cilium Cluster Mesh, Antrea Multi‑Cluster, and Kube‑OVN with ovn‑ic.

These solutions select gateway nodes within each cluster and establish tunnels between them, forwarding inter‑cluster traffic through those gateways.

Advantages

Self‑contained cross‑cluster capability without extra components.

Limitations

Works only with the same CNI across clusters; mixed‑CNI environments are unsupported. Requires careful CIDR planning to avoid overlapping address spaces. Only Cilium offers a relatively complete implementation with service discovery and network policies; others provide basic container networking. Again, a single blanket connection lacks fine‑grained control.

3. Submariner

Submariner is a CNI‑agnostic CNCF Sandbox project that connects clusters using a VXLAN‑based overlay. It selects gateway nodes, forwards traffic via VXLAN, and deploys CoreDNS for cross‑cluster service discovery while a Globalnet controller resolves CIDR overlap.

Advantages

Works across different CNI implementations. Provides cross‑cluster service discovery and DNS name resolution. Handles CIDR overlap, preventing IP conflicts when clusters interconnect.

Limitations

Not all CNI plugins expose traffic to the host, limiting gateway interception. Gateway nodes operate in active‑standby mode, lacking horizontal load‑balancing and potentially becoming a performance bottleneck. Still a blanket connection without fine‑grained traffic control.

4. Skupper

Skupper introduces a Virtual Application Network (VAN) that connects services at layer 7 instead of directly linking IP addresses. It builds a large message‑queue across clusters; packets are published to the queue and consumed by the remote side, effectively turning services into message topics.

Advantages

Fully CNI‑agnostic; operates at the application layer. Simple to get started, no complex network planning or CIDR constraints. Enables fine‑grained, on‑demand service‑level connectivity, reducing reliance on underlying network.

Limitations

Currently supports only TCP; UDP and lower‑level protocols like ICMP may fail. Message‑queue forwarding can lose original IP information. Potential latency and throughput penalties compared to direct networking. TCP stateful semantics may not map cleanly to a message‑queue model.

5. KubeSlice

KubeSlice, a recent CNCF Sandbox project, creates a dynamic secondary network interface for selected Pods. An overlay network runs on this interface, providing service discovery, DNS, QoS, network policies, and monitoring while preserving full protocol compatibility.

Advantages

Excellent CNI compatibility; the extra NIC isolates the overlay from the primary CNI. Full protocol compatibility because traffic is forwarded at layer 2. High flexibility: users can create multiple virtual networks per application need via CLI tools. Comprehensive feature set covering service discovery, DNS, QoS, network policies, and observability.

Limitations

Applications must be aware of the secondary NIC, potentially requiring code changes. Monitoring and tracing tools that rely on the primary Pod IP may need adaptation. Documentation is still evolving; many APIs and parameters lack clear explanations, making adoption harder for external users.

Conclusion

Multiple viable options exist for enabling efficient communication across Kubernetes clusters. Each approach offers distinct trade‑offs in terms of simplicity, CNI compatibility, protocol support, fine‑grained control, and operational overhead. Selecting the right solution depends on the specific requirements and constraints of your environment.