How Terway DataPath V2 Boosts Kubernetes Network Observability

Background

Container networking in ACK consists of three address spaces: Pod CIDR, Service CIDR and Node CIDR. The Pod CIDR and Node CIDR share the same address range. Each Pod has an eth0 interface with its own IP; the ENI auxiliary NIC is attached to the host, not directly to the Pod namespace. All traffic destined for a Pod IP is routed to the corresponding Calico virtual NIC (named cali<xx>) on the host.

Terway DataPath V2 Architecture

Terway can allocate 6‑20 auxiliary IPs per ENI, enabling high‑density Pod deployment. It supports two connectivity modes: veth‑pair routing and IPVLAN. Since Cilium v1.12 deprecated IPVLAN, Terway switched to DataPath V2 (unified data‑plane handling) starting with version v1.8.0.

Pod CIDR and node CIDR are identical.

Pod eth0 holds the Pod IP; the ENI auxiliary NIC is not placed inside the Pod namespace.

Traffic for a Pod IP is forwarded to the host’s Calico NIC ( cali<xx>).

Data‑Plane Forwarding Mechanics

Mapping between ENI auxiliary IPs and Pods can be displayed with:

kubectl -n kube-system exec -it terway-eniip-v5v2p -c terway -- terway-cli mapping

In DataPath V2, eBPF programs run on the host’s Calico NICs to accelerate both Pod‑to‑Pod and Pod‑to‑Service traffic, bypassing the host kernel stack whenever possible. Two eBPF redirect helpers are used:

bpf_redirect_peer : Sends packets directly to the destination Pod’s veth interface, avoiding the host LXC queue.

bpf_redirect_neigh : Populates source and destination MAC addresses for egress packets, skipping kernel routing.

Typical Traffic Scenarios

Eight representative scenarios were examined. Each scenario includes the environment, kernel routing tables and packet‑capture observations.

Same‑node Pod‑to‑Pod (including same‑node Service ClusterIP) – Traffic stays on the host, passes through the Calico NIC and is accelerated by eBPF (path: ECS Pod → cali<xx> → destination Pod).

Same‑node Pods on the same ENI – eBPF redirects traffic directly between the two Calico NICs without traversing the host network stack.

Same‑node Pods on different ENIs – eBPF still accelerates the path, but the host ENI NICs are involved before reaching the destination Calico NIC.

Cross‑node Pod‑to‑Pod – Traffic leaves the source ENI, traverses the VPC, passes through the destination ENI, and is accelerated by eBPF on both hosts.

In‑cluster Service (same ENI) – The Service IP is rewritten at the source Calico NIC to the backend Pod IP, bypassing the host stack.

In‑cluster Service (different ENIs on the same node) – Same rewrite logic as scenario 5, but the backend Pod resides on a different ENI.

In‑cluster Service (different nodes) – Combines the cross‑node path of scenario 4 with eBPF‑based Service IP translation.

External access to Service ExternalIP – External traffic reaches the Service via SLB. In older kernels (Aliyun 2) packets pass through the Pod’s Calico NIC; in newer kernels (Aliyun 3) eBPF on the host ENI NIC bypasses the Calico NIC entirely.

Observations

Kernels < 4.2 do not support eBPF acceleration. Aliyun 2 provides partial acceleration, while Aliyun 3 offers full acceleration.

Pod‑to‑Pod traffic generally avoids the host kernel stack when eBPF is enabled, reducing latency.

Service traffic is rewritten at the source Calico NIC, so the destination ENI never sees the original Service IP.

External SLB traffic may still traverse the Pod’s veth interface on older kernels but is fully accelerated on newer kernels.

Key Commands

# Show ENI‑auxiliary IP to Pod mapping
kubectl -n kube-system exec -it terway-eniip-v5v2p -c terway -- terway-cli mapping

References

https://docs.cilium.io/en/v1.12/operations/upgrade/#deprecated-options

https://help.aliyun.com/zh/ack/ack-managed-and-ack-dedicated/user-guide/ack-network-fabric-terway-eniip

https://help.aliyun.com/zh/ack/ack-managed-and-ack-dedicated/user-guide/work-with-terway

https://help.aliyun.com/zh/ack/ack-managed-and-ack-dedicated/user-guide/ack-network-fabric-terway-eni-trunking#RS9Nc