Mastering Kubernetes CNI: Choose the Right Network Plugin for Your Cloud‑Native Apps

This article introduces common container network application scenarios and the corresponding Kubernetes CNI plugin implementations, helping cloud‑native users quickly choose suitable network tools.

Common Network Plugins

We have heard of Docker bridge, Vethpair, VxLAN, and after moving to Kubernetes we learned Flannel, Calico, representing Overlay and Underlay modes. The article lists popular open‑source CNI plugins: Kube‑OVN, Antrea, Cilium.

Kube‑OVN and Antrea are based on OpenvSwitch; Cilium uses eBPF.

Deploying Kubernetes clusters with different CNI plugins can be done with Kubekey.

network:
  plugin: calico/kubeovn/cilium
  kubePodsCIDR: 10.233.64.0/18
  kubeServiceCIDR: 10.233.0.0/18

Antrea can be installed via Helm addon:

addons:
- name: antrea
  namespace: kube-system
  sources:
    chart:
      name: antrea
      repo: https://charts.antrea.io

One‑line command to create a cluster with Kubekey:

🐳 → kk create cluster --with-kubernetes --with-kubesphere

Network Application Scenarios

After having a Kubernetes cluster, consider scenarios beyond basic connectivity, such as fixed IP, network isolation, multi‑cluster interconnect, egress/ingress restrictions, bandwidth limits, and gateway access.

Fixed IP – needed for migrations where services are called by IP.

Network Isolation – tenants or apps should not communicate.

Multi‑cluster Network Interconnect – inter‑cluster pod communication.

Egress Restriction – control which pods can access external databases.

Ingress Restriction – limit external access to specific pods.

Bandwidth Limitation – throttle pod‑to‑pod traffic.

Outbound Gateway Access – SNAT for audit and security.

Network Plugin Feature Implementations

Fixed IP

Most CNI plugins have IPAM and support fixed IP via annotations.

Calico

"cni.projectcalico.org/ipAddrs": "[\"192.168.0.1\"]"

Kube‑OVN

ovn.kubernetes.io/ip_address: 192.168.100.100
ovn.kubernetes.io/ip_pool: 192.168.100.201,192.168.100.202

Antrea

Antrea IPAM works only in Bridge mode; with Multus, the primary interface uses NodeIPAM and the secondary uses Antrea IPAM.

ipam.antrea.io/ippools: 'pod-ip-pool1'
ipam.antrea.io/pod-ips: '<ip-in-pod-ip-pool1>'

Cilium

Not yet supported.

Multi‑cluster Network Interconnect

Calico can use BGP to exchange routes between clusters, but requires coordination with the physical network.

Overlay inter‑cluster connectivity can be achieved with VxLAN/GENEVE tunnels, supported by Kube‑OVN, Antrea, Cilium.

Submariner

Submariner creates gateway nodes in each cluster and establishes tunnels.

🐳 → subctl deploy-broker --kubeconfig ~/.kube/config1
🐳 → subctl join --kubeconfig ~/.kube/config1 broker-info.subm --clusterid ks1 --natt=false --cable-driver vxlan --health-check=false
🐳 → subctl join --kubeconfig ~/.kube/config2 broker-info.subm --clusterid ks2 --natt=false --cable-driver vxlan --health-check=false
🐳 → subctl show all

Cilium

Cilium Cluster Mesh can enable multi‑cluster connectivity.

🐳 → cilium clustermesh enable --context $CLUSTER1
🐳 → cilium clustermesh enable --context $CLUSTER2

Kube‑OVN

Kube‑OVN provides an OVN‑IC Docker container as a routing relay for inter‑cluster connectivity.

Multi‑cluster Service Inter‑Access

Submariner, Cilium, and Antrea support ServiceExport/ServiceImport for cross‑cluster services.

Submariner

Exports a Service in one cluster and imports it in another, creating a clusterset.local DNS domain.

Antrea

Uses ResourceExport/ResourceImport CRDs to achieve similar functionality.

Cilium

Uses Global Service annotations.

apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    io.cilium/global-service: 'true'
    io.cilium/shared-service: 'true'
    io.cilium/service-affinity: 'local'
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    name: nginx

Network Policies

Pod isolation, ingress/egress restrictions are expressed via Kubernetes NetworkPolicy, which works at L3/L4. Advanced capabilities (L7, node selectors, drop/reject) are provided by Cilium and Antrea.

Cilium

Cilium adds CRDs CiliumNetworkPolicy and CiliumClusterwideNetworkPolicy, supporting L3‑L7 policies, DNS filtering, HTTP method restrictions, and Kafka role‑based rules.

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "to-fqdn"
spec:
  endpointSelector:
    matchLabels:
      app: test-app
  egress:
  - toEndpoints:
    - matchLabels:
        "K8s:io.kubernetes.pod.namespace": kube-system
        "K8s:K8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
    rules:
      dns:
      - matchPattern: "*"
  - toFQDNs:
    - matchName: "my-remote-service.com"

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "rule"
spec:
  description: "L7 policy to restrict access to specific HTTP call"
  endpointSelector:
    matchLabels:
      org: empire
      class: deathstar
  ingress:
  - fromEndpoints:
    - matchLabels:
        org: empire
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      rules:
        http:
        - method: "POST"
          path: "/v1/request-landing"

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "rule1"
spec:
  description: "enable empire-hq to produce to empire-announce and deathstar-plans"
  endpointSelector:
    matchLabels:
      app: kafka
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: empire-hq
    toPorts:
    - ports:
      - port: "9092"
        protocol: TCP
      rules:
        kafka:
        - role: "produce"
          topic: "deathstar-plans"
        - role: "produce"
          topic: "empire-announce"

Antrea

Antrea provides AntreaNetworkPolicy and AntreaClusterNetworkPolicy, as well as Group and Tier concepts for hierarchical policy.

Example of rejecting ICMP ping:

apiVersion: crd.antrea.io/v1alpha1
kind: ClusterNetworkPolicy
metadata:
  name: acnp-reject-ping-request
spec:
  priority: 5
  tier: securityops
  appliedTo:
  - podSelector:
      matchLabels:
        role: server
    namespaceSelector:
      matchLabels:
        env: prod
  egress:
  - action: Reject
    protocols:
    - icmp:
        icmpType: 8
        icmpCode: 0
    name: DropPingRequest
    enableLogging: true

FQDN based filtering example:

apiVersion: crd.antrea.io/v1alpha1
kind: ClusterNetworkPolicy
metadata:
  name: acnp-fqdn-all-foobar
spec:
  priority: 1
  appliedTo:
  - podSelector:
      matchLabels:
        app: client
  egress:
  - action: Allow
    to:
    - fqdn: "*foobar.com"
    ports:
    - protocol: TCP
      port: 8080
  - action: Drop
    enableLogging: true

Egress

For pods that need to access external services via a fixed egress IP or SNAT, Cilium, Kube‑OVN, and Antrea provide egress gateway policies.

Cilium

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: egress-sample
spec:
  selectors:
  - podSelector:
      matchLabels:
        app: snat-pod
        io.kubernetes.pod.namespace: default
  destinationCIDRs:
  - "0.0.0.0/0"
  egressGateway:
    nodeSelector:
      matchLabels:
        node.kubernetes.io/name: node1
    egressIP: 10.168.60.100

Kube‑OVN

apiVersion: v1
kind: Pod
metadata:
  name: pod-gw
  annotations:
    ovn.kubernetes.io/eip: 172.10.0.1
spec:
  containers:
  - name: eip-pod
    image: nginx:alpine

Antrea

apiVersion: crd.antrea.io/v1alpha2
kind: Egress
metadata:
  name: egress-staging-web
spec:
  appliedTo:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: staging
    podSelector:
      matchLabels:
        app: web
  externalIPPool: external-ip-pool

Bandwidth Management

Kube‑OVN and Cilium support bandwidth limits; Kube‑OVN also supports QoS via annotations.

Kube‑OVN

apiVersion: v1
kind: Pod
metadata:
  name: qos
  namespace: ls1
  annotations:
    ovn.kubernetes.io/ingress_rate: "3"
    ovn.kubernetes.io/egress_rate: "1"
    ovn.kubernetes.io/latency: 3
    ovn.kubernetes.io/loss: 20

Cilium

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/egress-bandwidth: 10M

Conclusion: From early Docker networking to the modern CNCF CNI ecosystem, cloud‑native networking has evolved to provide connectivity, security, observability, and advanced features for operators.