How Containers Communicate in Kubernetes: Network Basics and CNI Explained

Container Network Basics

In Kubernetes, network connectivity between containers is essential, but Kubernetes itself does not implement a container network; it relies on plug‑in mechanisms. The basic principles for a container network are:

Pods on any node can communicate directly without NAT.

Nodes and Pods can communicate, and Pods can reach any network.

Each Pod has its own network stack; the address seen inside the Pod is the same as seen from outside, and all containers in a Pod share that stack.

Linux Network Namespace and Veth Pair

A Linux container’s network stack is isolated in its own network namespace, which contains a network interface, loopback device, routing table and iptables rules. Implementing container networking relies on several Linux features:

Network namespaces isolate protocol stacks.

Veth pairs provide a pair of virtual Ethernet devices that connect two namespaces.

Iptables/Netfilter handle packet filtering and NAT.

Linux bridge acts as a layer‑2 virtual switch.

Routing tables determine packet forwarding.

When two containers share the same host, they can be viewed as two hosts connected by a virtual cable; the Docker bridge docker0 and a veth pair enable their communication.

Example: Inspecting a Container’s Network

docker run -d --name c1 hub.pri.ibanyu.com/devops/alpine:v3.8 /bin/sh

Inside the container:

docker exec -it c1 /bin/sh
# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
...
# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.17.0.1      0.0.0.0         UG    0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0

The eth0 interface is one end of the veth pair; the default route points to the Docker bridge.

# brctl show
docker0  8000.02426a4693d2  no  veth20b3dac

On the host, the other end of the veth pair appears as veth20b3dac attached to docker0.

Pod‑to‑Pod Communication on the Same Host

docker run -d --name c2 -it hub.pri.ibanyu.com/devops/alpine:v3.8 /bin/sh
docker exec -it c1 /bin/sh
# ping 172.17.0.3

The ping succeeds because the destination IP matches a direct route; the ARP broadcast is forwarded by the Docker bridge to the appropriate veth interface.

Cross‑Host Network Communication

By default, containers on different hosts cannot reach each other by IP. Kubernetes uses the CNI (Container Network Interface) API to invoke plug‑ins that create a host‑wide bridge (usually cni0) and configure pod network stacks.

CNI plug‑ins implement three models:

Overlay: encapsulates the entire pod network over the underlying network (e.g., Flannel with VXLAN, Calico IPIP).

Routing (host‑gw): uses the host’s routing table to reach pod subnets; requires the hosts to be on the same L2 network (e.g., Flannel host‑gw).

Underlay: relies on the underlying L3 network; pods remain in separate subnets but communicate via normal routing (e.g., Calico BGP).

Example of Flannel host‑gw routing rule: 10.244.1.0/24 via 10.168.0.3 dev eth0 Flannel updates each node’s routing table so that traffic to a pod subnet is sent to the node that owns that subnet.

Calico Architecture

Calico consists of:

Calico CNI plug‑in for Kubernetes integration.

Felix, which programs host routing rules and forwarding information.

BIRD, which distributes routes via BGP.

Confd for configuration management.

Calico does not create a bridge; instead it adds a veth pair for each pod and installs a route like: 10.92.77.163 dev cali93a8a799fe1 scope link When hosts are not on the same L2 network, Calico can use IPIP encapsulation (overlay) or rely on an underlay network with BGP‑learned routes.

In a large cluster, Calico’s default node‑to‑node mesh creates O(N²) BGP sessions; for clusters larger than ~50 nodes, a router‑reflector (RR) topology is recommended.

Choosing a CNI Plug‑in

In public‑cloud environments, the cloud provider’s CNI or Flannel host‑gw is often sufficient. In private data‑center deployments, Calico’s BGP‑based routing provides better performance and flexibility. Select the plug‑in that matches your network topology and performance requirements.

Original source: https://tech.ipalfish.com/blog/2020/03/06/kubernetes_container_network/