How to Access Kubernetes ClusterIP Services and Pods from Outside the Cluster

Background and Challenge

A JD Cloud middle‑platform engineer received a request to break the default Kubernetes limitation that prevents direct access to ClusterIP services and pods from an external office network. The customer needed to reach the service IP and the backend pod directly, which is normally only reachable inside the cluster.

Service Types Overview

Kubernetes provides several service types:

ClusterIP – reachable only within the cluster.

NodePort – maps a port on every node to the service, allowing external access via node IP.

LoadBalancer – creates a cloud load‑balancer that forwards traffic to the service.

Because the managed K8s product does not support binding public IPs to nodes,

NodePort</strong> was not feasible. The recommended approach was to use <strong>LoadBalancer</strong> services, but the customer had hundreds of services, making it impractical to create a separate LB for each.</p><h2>Network Architecture and Connectivity Options</h2><p>The customer's office network connects to the cloud via a VPN or dedicated line. The K8s VPC consists of three subnets: <em>node</em> (node communication), <em>NAT/LB</em> (for NAT gateways and load balancers), and <em>pod</em> (pod communication). Pods obtain IPs from elastic network interfaces attached to each node.</p><p>Two connectivity methods were considered:</p><ol><li>Dedicated line (BGW) – cannot route to specific nodes, so it cannot reach ClusterIP services.</li><li>Self‑built IPsec VPN – a cloud host in the same VPC acts as the VPN endpoint. Traffic can be routed to a specific node, then to the pod subnet.</li></ol><h2>Implementation and Testing</h2><p>A test environment was built with two cloud hosts: one simulating the office side (Shanghai) and one in the K8s VPC (Beijing). Routes were configured so that the service’s next‑hop pointed to a node (<strong>node A</strong>). Initial tests showed that requests to an <strong>nginx</strong> service succeeded, but a <strong>MySQL</strong> service on a different node (<strong>node B</strong>) failed.</p><p>Packet captures (<code>tcpdump

) revealed that when the service and pod reside on the same node, the return traffic follows the expected path. When they are on different nodes, the pod’s reply is sent to the VPN host instead of the forwarding node, causing the cloud platform to drop the packet as a stray reply.

Root Cause Analysis

The failure stemmed from asymmetric routing: the pod’s reply left the cluster via the eth1 interface of node A, which, due to the pod subnet’s routing table, was forwarded to the VPN host rather than back to the forwarding node. Consequently, the client never received the response.

Solution – Enabling Masquerade‑All

To hide the pod’s original IP and force all traffic to appear as if it originates from the service IP, the --masquerade-all flag of kube‑proxy was enabled. This activates SNAT for service traffic, ensuring that both request and response packets are rewritten to use the service IP, eliminating the asymmetric routing problem.

Steps:

Edit the kube-proxy ConfigMap in the kube-system namespace and set masqueradeAll: true.

Delete existing kube-proxy pods; the DaemonSet recreates them with the new configuration.

After applying the change, requests to services on different nodes succeeded, and direct pod access also worked.

Verification

Post‑change tests using curl and paping confirmed successful HTTP access to the nginx service and MySQL port 3306. Packet captures showed that SNAT rewrote source and destination IPs, so the pod never directly contacts the client.

Conclusion

Enabling --masquerade-all resolves the external access issue for ClusterIP services by performing SNAT on both directions. While this meets the immediate requirement, exposing ClusterIP services outside the cluster is not recommended for production due to security and performance considerations. The case study also deepens understanding of Kubernetes service networking, kube‑proxy modes (iptables vs. IPVS), and the impact of SNAT.

Key Takeaways

ClusterIP services are internal‑only; external access requires careful routing or SNAT.

Asymmetric routing can cause packet loss when pods reside on different nodes.

Enabling --masquerade-all forces SNAT, making the service act as a true proxy.

Always evaluate security implications before exposing internal services.