Decoding Terway ENI‑Trunking: Data‑Plane Paths and SOP Scenarios in Alibaba Cloud

In recent years, enterprises have accelerated the cloud‑native transformation of their infrastructure, moving from simple IaaS to micro‑service architectures that demand fine‑grained networking and observability. To lower the high barrier of observing cloud‑native networks, ACK product research and the Alibaba Elastic Service (AES) jointly built the ack net‑exporter and a series of cloud‑native network data‑plane observability guides.

Container Network Overview

The container network consists of three logical segments: the Pod CIDR, the Service CIDR, and the Node CIDR. These segments must interconnect while enforcing access control, and the choice of underlying technology (Flannel, Terway, etc.) determines performance and scalability.

Terway ENI‑Trunking Architecture

Elastic Network Interface (ENI) Trunking attaches a virtual NIC to a VPC‑type ECS instance. Compared with a regular ENI, a Trunk ENI dramatically increases the instance’s pod‑density. When the Trunk ENI feature is enabled, selected Pods are bound to a Trunk ENI; other Pods continue to use the shared ENI. Pods that explicitly request custom configuration receive a dedicated ENI (Member ENI) together with a dedicated virtual switch and security group.

Key technical points:

Each Pod receives an IP from the auxiliary IP pool of its Member ENI.

A custom PodNetworking CRD describes the desired network plane; Terway creates a corresponding PodENI resource to track the allocated ENI.

Traffic between the host and the Pod traverses a veth pair linked to the Calico virtual NIC (cali*). The host’s routing table forwards Pod‑IP traffic to the appropriate cali interface, which then routes to the Member ENI.

VLAN tags are applied/removed at the tc layer (e.g., VLAN ID 1027) to steer traffic from the Trunk ENI to the correct Member ENI.

Security groups can be attached per‑Pod, per‑ENI, enabling fine‑grained isolation that satisfies strict compliance requirements in finance, telecom, and government sectors.

Overall ENI‑Trunking Benefits

Higher pod density through shared Trunk ENI while still supporting per‑Pod IP, switch, and security‑group configuration.

Requires ECS instances of newer generations (5th/6th gen, 8 CPU+), with Trunk ENI support.

Maximum pod count per node = (ECS ENI count – 1) × IP‑per‑ENI.

Pod‑level security‑group rules do not affect intra‑node pod‑to‑pod traffic; NetworkPolicy can be used for intra‑node isolation.

Traffic steering relies on VLAN IDs to map a Pod’s flow to its Member ENI.

10 SOP Scenarios for ENI‑Trunking

2.1 Scenario 1 – Node‑to‑Pod Access (same or different security groups)

Environment: Node cn‑hongkong.10.0.4.22 runs nginx‑6f545cb57c‑kt7r8 (IP 10.0.4.30). The host’s default route points to eth0, while the Pod’s traffic is routed through the Calico cali interface and the associated veth pair. The Pod is bound to a Member ENI and a Trunk ENI, and VLAN ID 1027 is observed on egress/ingress.

Result: The node can reach the Pod; security‑group rules on the ENI determine allowed ports.

2.2 Scenario 2 – Same‑Node, Same‑Security‑Group Trunk Pod Mutual Access

Two Pods ( nginx and busybox) on the same node each have a matching PodNetworking configuration, receiving distinct Member ENIs but sharing the same security group. Traffic again passes through the Calico NIC and VLAN tagging.

Result: Mutual access succeeds; security‑group permits traffic on port 80.

2.3 Scenario 3 – Same‑Node, Different‑Security‑Group Trunk Pods

Pods on node cn‑hongkong.10.0.4.244 ( nginx‑96bb9b7bb‑wwrdm and centos‑648f9999bc‑nxb6l) are bound to different security groups. The ENI‑level security groups restrict inbound traffic accordingly.

Result: Access is possible only when the source IP matches the allowed CIDR of the destination Pod’s security group.

2.4 Scenario 4 – Different‑Node, Same‑Security‑Group Trunk Pods

Client Pod on cn‑hongkong.10.0.4.20 communicates with server Pod on cn‑hongkong.10.0.4.22. Both Pods belong to the same security group, so the VPC routing and VLAN‑based forwarding allow traffic to traverse the Trunk ENI, Member ENI, and Calico layers.

Result: Communication succeeds; security‑group symmetry is required.

2.5 Scenario 5 – Different‑Node, Different‑Security‑Group Trunk Pods

Client and server Pods reside on separate nodes and belong to distinct security groups. The traffic path includes host‑level routing, VPC route rules, Trunk ENI, and Member ENI. Whether the connection succeeds depends entirely on the cross‑security‑group rules.

Result: Access is conditional on explicit allow rules in both security groups.

2.6 Scenario 6 – In‑Cluster Source → Service (ClusterIP / ExternalIP, same security group, Local mode)

Client Pod on cn‑hongkong.10.0.4.20 accesses a Service whose ClusterIP is 192.168.81.92 and ExternalIP is 8.210.162.178. The request is NAT‑ed by ipvs, leaves the host via eth0, traverses the Member ENI of the destination Pod, and returns through the same path. All three ENIs (host eth0, source Member ENI, destination Member ENI) affect connectivity.

Result: Connectivity hinges on security‑group rules for all involved ENIs.

2.7 Scenario 7 – In‑Cluster Source → Service (different security groups, Local mode)

Similar to Scenario 6 but source and destination Pods belong to different security groups. The packet flow is identical, but the security‑group policies of both ENIs must allow bidirectional traffic.

Result: Successful communication requires mutual security‑group allowances.

2.8 Scenario 8 – Cluster‑Mode Service Access (ExternalIP, different security groups)

When ExternalTrafficPolicy=Cluster, the request may be load‑balanced across nodes. The traffic still follows the host‑→ Calico → VPC route → Member ENI path, with VLAN tagging. Security groups on both source and destination ENIs, as well as any intermediate node ENIs, determine reachability.

Result: Access is possible only if all involved security groups permit the traffic.

2.9 Scenario 9 – Cluster‑Mode Service Access (ExternalIP, same security group)

Both source and destination Pods share the same security group, simplifying the rule set. The data path remains host → Calico → VPC → Member ENI, with ipvs NAT. All ENIs involved must have the security group open for the pod IP range.

Result: Communication succeeds when the shared security group allows the pod IP ranges.

2.10 Scenario 10 – External Client → Service IP

External traffic reaches the Service via an SLB. The SLB forwards the request to the Member ENIs of the target Pods (the ENIs are part of the SLB backend pool). The packet then traverses the Trunk ENI, VLAN tag, and finally the Pod’s eth0. The SLB’s ExternalTrafficPolicy (Local or Cluster) determines whether traffic is sent to a single node or load‑balanced across nodes.

Result: Access depends on both SLB backend ENI security groups and the Pod’s security group.

Conclusion

The article consolidates the ten SOP scenarios of ACK’s Terway ENI‑Trunking mode, illustrating how VLAN‑based routing, per‑Pod ENI allocation, and security‑group configuration jointly enable high‑density, fine‑grained network management for cloud‑native workloads. Understanding these data‑plane paths helps troubleshoot network jitter, optimize configurations, and prepare for the next series on Terway ENIIP and ASM Istio.