Unlocking Edge Cloud: Kube-OVN’s Security Groups, VIP, Live Migration & Smart NIC Offload

1. Access Control: Security Groups

The edge‑cloud team initially considered using native Kubernetes NetworkPolicy for access control but rejected it due to limited granularity (no ICMP, ARP/DHCP protection, rule priority) and poor fit for IaaS users accustomed to traditional security groups.

They first implemented security groups for VMs on KubeVirt using iptables, mirroring early OpenStack designs. This approach could not protect containers and broke when smart‑NIC offload bypassed the kernel.

To overcome these limits, the team switched to OVS CT flow tables and introduced a custom SecurityGroup CRD in Kube‑OVN. The new implementation supports both containers and VMs, works with hardware offload, and aligns with OpenStack feature parity. See PR https://github.com/kubeovn/kube-ovn/pull/932 and the Chinese wiki https://github.com/fanriming/kube-ovn/wiki/安全组 for details.

2. High‑Availability Virtual IP

Traditional HA solutions rely on virtual IPs (VIP) managed by keepalived or dedicated load‑balancers, which public clouds restrict for security reasons. Kube‑OVN adds a VIP feature that works when port_security is enabled, ensuring packets with non‑allocated IPs are dropped.

Usage steps:

Reserve an IP in a Subnet by setting the excludeIps field.

Bind the reserved IP to a pod via the annotation ovn.kubernetes.io/vip (or similar) on the target pod.

The annotation can be updated dynamically, allowing VIP reclamation or reassignment.

Implementation reference: PR https://github.com/kubeovn/kube-ovn/pull/1036.

3. Live Migration Support

While KubeVirt provides basic VM live migration, standard CNI designs prevent seamless network continuity. Kube‑OVN addresses this by keeping the pod’s default network for migration traffic and attaching the VM’s business network via Multus.

Key points:

Use a Multus network attachment for the VM’s data plane, preserving the default pod network for migration.

Annotate the VM with

<attach>.<ns>.ovn.kubernetes.io/allow_live_migration

to avoid IP‑conflict errors.

If DHCP is used, select the default route interface with the annotation ovn.kubernetes.io/default_route.

The solution satisfies KubeVirt’s migration requirements (ports 49152‑49153 availability, no bridge‑type pod network) and is demonstrated in PR https://github.com/kubeovn/kube-ovn/pull/1001.

4. Smart NIC Hardware Offload

Edge workloads demand both ultra‑low latency and SDN capabilities. The team integrated Mellanox CX5 smart NIC offload into Kube‑OVN, achieving millions of packets per second.

Challenges and resolutions:

OVS version compatibility – Kube‑OVN v1.6+ uses OVS 2.15, but offload requires ≤2.14. The solution pins OVN 20.06 with OVS 2.14.

LB feature uses dp_hash flow tables not supported by CX5; a toggle enable-lb=false disables the offload‑incompatible path.

Stateful CT flows for NetworkPolicy and security groups need a recent kernel (Linux 5.12) and Mellanox OFED driver (MLNX_OFED_LINUX‑5.4‑1.0.3.0).

VMs use SR‑IOV VF interfaces; the NetworkAttachmentDefinition includes vf_driver: vfio-pci so Kube‑OVN CNI can prepare the VF, while KubeVirt attaches it via standard SR‑IOV configuration.

After these adaptations, the stack (Smart NIC + Kube‑OVN + KubeVirt) delivers both conventional SDN functions and high‑performance edge networking.

Conclusion

The 2022 work of the China Telecom Tianyi Cloud edge‑computing team demonstrates how community contributions can evolve Kube‑OVN to meet demanding edge scenarios, adding security groups, VIP, live‑migration‑ready networking, and smart‑NIC offload. Future plans aim to further enrich the network capabilities built on Kube‑OVN.