How Do Packets Reach Pods? Unveiling Kubernetes Service & Calico Routing

1. Overview

We expose Kubernetes services externally in several ways; one is using an external‑IPs ClusterIP service with kube‑proxy in iptables mode. External IPs point to specific worker nodes that act as traffic‑forwarding nodes, and the service is reachable via vip:port, distinguishing business by port.

Our production clusters use Calico as the CNI plugin, deployed peered with TOR (Top of Rack) routers . Each worker node establishes a BGP peer with its top‑of‑rack switch, which in turn peers with core switches, allowing pod IPs to be reachable directly within the corporate network.

The core question is: how does a packet, arriving at a node_ip:port or cluster_ip:port, find the pod IP and finally the pod itself?

2. Principle Analysis

When a packet reaches the worker node (e.g., node A) via the service VIP, kube‑proxy’s iptables rules DNAT the destination to the pod IP. Because Calico’s BGP deployment distributes routes for the pod CIDR to the core and top‑of‑rack switches, the packet is then routed to the appropriate worker node (e.g., node B) that hosts the target pod. On that node, Calico’s pre‑installed routing rules and the virtual veth pair move the packet from the host network namespace to the pod network namespace.

To try it locally, start a Minikube cluster with Calico:

minikube start --network-plugin=cni --cni=calico
# or
minikube start --network-plugin=cni
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Deploy a sample nginx deployment with two replicas and expose it via a ClusterIP service with ExternalIPs and a NodePort service:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-demo-1
  labels:
    app: nginx-demo-1
spec:
  replicas: 2
  template:
    metadata:
      name: nginx-demo-1
      labels:
        app: nginx-demo-1
    spec:
      containers:
      - name: nginx-demo-1
        image: nginx:1.17.8
        imagePullPolicy: IfNotPresent
        livenessProbe:
          httpGet:
            port: 80
            path: /index.html
          failureThreshold: 10
          initialDelaySeconds: 10
          periodSeconds: 10
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-demo-1
spec:
  selector:
    app: nginx-demo-1
  ports:
  - port: 8088
    targetPort: 80
    protocol: TCP
  type: ClusterIP
  externalIPs:
  - 192.168.64.57   # replace with your worker node IP
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-demo-2
spec:
  selector:
    app: nginx-demo-1
  ports:
  - port: 8089
    targetPort: 80
  type: NodePort
---

After deployment, the service can be accessed via the ExternalIP ClusterIP service or the NodePort service.

3. iptables Custom Rules

When a packet arrives at node_ip:port or cluster_ip:port, the kernel DNATs it to the pod IP and SNATs the return traffic. Calico’s documentation illustrates the flow for ClusterIP and NodePort services.

Kube‑proxy creates a custom KUBE‑SERVICES chain in the nat table, which is invoked in the PREROUTING chain. The relevant rules can be listed with:

sudo iptables -v -n -t nat -L PREROUTING | grep KUBE-SERVICES
sudo iptables -v -n -t nat -L KUBE-SERVICES
sudo iptables -v -n -t nat -L KUBE-NODEPORTS

Both internal (cluster_ip:port) and external (external_ip:port or node_ip:port) accesses match a KUBE‑SVC‑… chain that forwards to a KUBE‑SEP‑… chain. Because the deployment has two pod replicas, the traffic is load‑balanced with a 50 % round‑robin algorithm using iptables’ statistic module, eventually DNAT‑ed to a specific pod IP (e.g., 10.217.120.72:80).

sudo iptables -v -n -t nat -L KUBE-SVC-JKOCBQALQGD3X3RT
sudo iptables -v -n -t nat -L KUBE-SEP-CRT5ID3374EWFAWN
sudo iptables -v -n -t nat -L KUBE-SVC-6JCCLZMUQSW27LLD
sudo iptables -v -n -t nat -L KUBE-SEP-SRE6BJUIAABTZ4UR

4. Calico Routing and Virtual Interfaces

Calico’s BGP deployment ensures that pod CIDR routes are advertised to the network switches. For example, a route entry like 10.20.30.40/26 via 10.203.30.40 tells the network that the pod subnet is reachable via a specific worker node.

# Example route entry (illustrative)
Network               NextHop   ...
10.20.30.40/26        10.203.30.40   ...

On the worker node that hosts the target pod, the host network namespace shows a virtual interface (e.g., cali1087c975dd9) with an index that matches the veth pair inside the pod (e.g., eth0). The index can be inspected by entering the pod’s network namespace using a netshoot container:

# Find the nginx container ID
docker ps -a | grep nginx
export CONTAINER_ID=f2ece695e8b9   # example ID
# Run netshoot in the same network and PID namespace
docker run -it --network=container:$CONTAINER_ID --pid=container:$CONTAINER_ID --ipc=container:$CONTAINER_ID nicolaka/netshoot:latest ip -c addr

The routing table and virtual interface are created by Calico’s network plugin, while pod IP allocation is managed by Calico’s IPAM plugin and stored in the Calico datastore.

5. Summary

Regardless of whether the service is accessed via cluster_ip:port, external_ip:port, or node_ip:port, the packet is first processed by kube‑proxy’s iptables rules, which DNAT it to the appropriate pod IP. Calico’s BGP routing then directs the packet to the worker node that hosts the pod, and the node’s routing table and virtual veth pair move the packet from the host namespace into the pod’s network namespace.

References

https://docs.projectcalico.org/about/about-kubernetes-service

https://mp.weixin.qq.com/s/bYZJ1ipx7iBPw6JXiZ3Qu

https://mp.weixin.qq.com/s/oaW87xLnlUYYrwVjBnqee

https://mp.weixin.qq.com/s/RziLRPYqNoQEQuncm47rHg