Why HostPort Breaks Kubernetes Service Routing and How to Fix It

Problem Background

The cluster runs K8s v1.15.9 with flannel‑vxlan, portmap CNI and ipvs kube‑proxy. Two MySQL instances (Mysql‑A on node‑1, Mysql‑B on node‑2) are exposed via separate Services. Accessing Mysql‑A from node‑2 results in "authentication failed", while using Mysql‑B credentials actually connects to Mysql‑B, indicating traffic is being redirected.

Investigation

Comparing iptables on node‑1 and node‑2 revealed extra NAT rules on node‑2 that DNAT all traffic destined for port 3306 to the pod IP of Mysql‑B (10.224.0.222). These rules cause the observed redirection.

Root Cause

The extra rules are generated by the CNI portmap plugin when a pod specifies hostPort. The plugin inserts its rules at the front of the nat chains, giving them higher priority than the standard KUBE‑SERVICES rules (including NodePort). Consequently, hostPort traffic always wins.

The CNI ‘portmap’ plugin, used to set up HostPorts, inserts rules at the front of the iptables nat chains, which take precedence over the KUBE‑SERVICES chain.

Reproduction

Three nginx Deployments were created, two of them using hostPort on different ports. The resulting iptables rules matched the hostPort settings, confirming the behavior.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-hostport2
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: nginx-hostport2
  template:
    metadata:
      labels:
        k8s-app: nginx-hostport2
    spec:
      nodeName: spring-38
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80
              hostPort: 31123

Port Occupancy

HostPort rules are visible only via iptables, not via ipvsadm, lsof, or netstat. The scheduler prevents placing another pod with the same hostPort on the same node, but if forced, deployment fails with errors.

Resolution

Removing the hostPort specification from the pod spec automatically deletes the injected iptables rules. For production environments, avoid using hostPort unless absolutely necessary.

Production Recommendation

Do not use hostPort in production; it interferes with scheduling and can cause traffic to be misrouted, leading to severe issues.