Why Can’t Pods Reach CLB on Port 443? Unraveling K8s IPVS and TrafficPolicy Issues

When a pod tries to reach a Cloud Load Balancer (CLB) on port 443, the request fails with Connection refused, while the same CLB on port 80 works; external ECS nodes can reach both ports. The root cause is that the CLB IP (e.g., 192.168.1.200) is bound to the dummy interface kube‑ipvs0 on each Kubernetes node. Because the Service SVC1 is of type LoadBalancer, kube‑proxy creates IPVS rules only for the ports defined in the Service (port 80 in this case). No IPVS rule exists for port 443, so traffic from pods is dropped before leaving the node.

If a node lacks an IPVS rule for the target port but a local process is listening on 0.0.0.0:443 , the node will answer with the local service rather than the real CLB backend.

To solve the problem, the article recommends separating internal and external traffic by using two distinct CLBs, or applying the Alibaba Cloud Service annotation

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname

to prevent the CLB IP from being bound to the dummy interface. This annotation requires CCM version v2.3.0 or later.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname: "${your_service_hostname}"
  name: nginx-svc
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

The article then delves into how externalTrafficPolicy (Cluster vs. Local) and the newer internalTrafficPolicy (default enabled in K8s 1.22) affect source‑IP preservation. Before Kubernetes 1.24, with externalTrafficPolicy=Local, nodes that lacked an endpoint for the Service would still create empty IPVS backends, causing the request to be rejected. Starting with 1.24, nodes without an endpoint mount all endpoints, resulting in SNAT and loss of the original source IP.

Detailed observations are presented for various CNI plugins (Flannel, Terway ENI, ENIIP, ENI‑Trunking, ASM Istio) and network modes (hostNetwork, ENIIP). Tables of behavior illustrate:

Cluster‑policy traffic always undergoes SNAT, losing the client IP.

Local‑policy traffic preserves the client IP only on nodes that host the Service endpoint.

After Kubernetes 1.24, Local‑policy on nodes without an endpoint still creates IPVS rules for all endpoints, leading to SNAT.

For NodePort and CLB access inside the cluster, the article shows concrete packet‑capture (conntrack) results confirming which node performs DNAT and which pod returns the response. It also highlights that accessing a CLB from a node without an endpoint fails because the CLB IP is occupied by the dummy interface and no SNAT is performed.

Finally, the article provides reference links to Alibaba Cloud documentation, the Service annotation guide, the Kubernetes service‑traffic‑policy page, and the relevant GitHub pull‑request that introduced the 1.24 behavior change.