Why HostPort Can Hijack Your Service Traffic in Kubernetes

Problem Background

In a Kubernetes v1.15.9 cluster using flannel‑vxlan and portmap CNI with ipvs mode, two MySQL instances (mysql‑A on node‑1 and mysql‑B on node‑2) exhibited strange behavior: accessing mysql‑A from node‑2 returned authentication errors, while using mysql‑B credentials succeeded and actually connected to mysql‑B.

Investigation Process

Since mysql‑A worked from node‑1, the issue was isolated to node‑2. Comparing iptables on node‑2 with node‑1 revealed extra NAT rules on node‑2 that redirected all traffic destined for port 3306 to the pod IP of mysql‑B (10.224.0.222).

-A CNI-DN-xxxx -p tcp -m tcp --dport 3306 -j DNAT --to-destination 10.224.0.222:3306
-A CNI-HOSTPORT-DNAT -m comment --comment "dnat name": \"cni0\" id: \"xxxxxxxxxxxxx\" -j CNI-DN-xxx
-A CNI-HOSTPORT-SNAT -m comment --comment "snat name": \"cni0\" id: \"xxxxxxxxxxxxx\" -j CNI-SN-xxx
-A CNI-SN-xxx -s 127.0.0.1/32 -d 10.224.0.222/32 -p tcp -m tcp --dport 80 -j MASQUERADE

These rules caused any request to port 3306 to be DNAT‑ed to mysql‑B, explaining the observed authentication behavior.

Root Cause

The extra rules were generated by the CNI portmap plugin when a pod used hostPort. The plugin inserts its rules at the front of the iptables NAT chains, giving them higher priority than the standard KUBE‑SERVICES chain.

The CNI ‘portmap’ plugin, used to set up HostPorts, inserts rules at the front of the iptables nat chains; these take precedence over the KUBE‑SERVICES chain, so HostPort rules can match before NodePort rules.

Thus, enabling hostPort caused the portmap plugin to add NAT rules that hijacked traffic intended for other services.

Reproduction

A test deployment with three Nginx pods was created; two used hostPort on different ports, and one used a regular Service. The experiment confirmed that hostPort rules were present in iptables but invisible to ipvsadm, lsof, or netstat, because they do not create a listening socket on the host.

Port Occupancy

When another pod attempted to use the same hostPort, the scheduler rejected the pod, demonstrating that hostPort influences scheduling. If forced onto the same node, the pod would fail to start, and the error messages accumulate.

Interaction with NodePort

Creating a NodePort Service on the same port as a hostPort showed that traffic to that port is still handled by the hostPort rule, because the hostPort NAT rule appears before the KUBE‑NODE‑PORT rule in the chain.

Solution

Removing the hostPort specification from pod specs eliminates the offending iptables rules. Alternatively, avoid using hostPort in production unless absolutely necessary, as it can interfere with scheduling and cause unexpected traffic redirection.

Production Recommendation

Do not use hostPort in production environments unless there is no other viable solution, because it affects pod scheduling and can lead to severe networking issues.

Source: https://izsk.me/2021/08/01/Kubernetes-hostport/