How to Preserve Client Source IP in Istio Service Mesh – Strategies and Configurations
This article explains why keeping the client source IP in an Istio service mesh matters, outlines the reasons for IP loss, and provides step‑by‑step solutions for north‑south and east‑west traffic using Service settings, traffic policies, and Envoy headers.
Importance of Preserving Source IP
Keeping the client source IP is essential for access‑control policies, load‑balancing decisions, and accurate data‑analysis logs.
What Preserving Source IP Means
It refers to preventing the real client IP from being replaced when a request passes through a load balancer or reverse proxy.
How to Identify Client Source IP
In Istio, the Envoy sidecar adds the
x-forwarded-forheader to HTTP requests. The steps are:
Check the x-forwarded-for header : it contains the IPs of all proxies on the request path.
Select the last IP : usually the IP closest to the server.
Validate trust : ensure the proxy that added the IP is trusted.
Use x-envoy-external-address if Envoy sets it, which holds the true client IP.
See the Envoy documentation for details about the
x-forwarded-forheader.
Test Environment
GKE
Client Version: v1.28.4
Kustomize Version: v5.0.4‑0.20230601165947‑6ce0bf390ce3
Server Version: v1.27.7‑gke.1121000
Istio
client version: 1.20.1
control plane version: 1.20.1
data plane version: 1.20.1 (12 proxies)
CNI
cilium‑cli: v0.15.18 compiled with go1.21.5 on darwin/amd64
cilium image (default): v1.14.4
cilium image (running): 1.14.5
North‑South Traffic
When a client outside the Kubernetes cluster accesses a service through a load balancer, the source IP is often replaced by the node IP (e.g.,
10.128.0.54).
Setting the Service
externalTrafficPolicyto
Localdisables kube‑proxy load‑balancing on the node, allowing the pod to see the original client IP.
<code>kubectl patch svc istio‑ingressgateway -p '{"spec":{"externalTrafficPolicy":"Local"}}' -n istio‑system</code>After applying the patch, a curl test shows the client IP
123.120.247.15correctly.
East‑West Traffic
For intra‑mesh traffic, the default Istio configuration also hides the client IP. Switching the sidecar interception mode to
TPROXYpreserves the source IP.
<code>kubectl patch deployment -n default echo‑server -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/interceptionMode":"TPROXY"}}}}'</code>After redeploying, a curl from the
sleeppod returns the pod IP
10.32.3.202as the client IP.
Multiple Proxy Layers
If traffic passes through several proxies before reaching the Istio mesh, configure
numTrustedProxiesin the gateway topology to indicate how many proxy hops are trusted. Envoy then extracts the correct client IP from the
x-forwarded-forheader.
<code>kubectl patch deployment istio‑ingressgateway -n istio‑system -p '{"spec":{"template":{"metadata":{"annotations":{"proxy.istio.io/config":"{\"gatewayTopology\":{\"numTrustedProxies\":2}}"}}}}}'</code>TCP Traffic
For L4 traffic, the HTTP‑specific
x-forwarded-forheader is unavailable. Use the Proxy Protocol to transmit the original client IP in the TCP handshake.
<code>kubectl patch deployment istio‑ingressgateway -n istio‑system -p '{"spec":{"template":{"metadata":{"annotations":{"proxy.istio.io/config":"{\"gatewayTopology\":{\"proxyProtocol\":{}}}"}}}}}'</code>Note that not all cloud load balancers support Proxy Protocol (e.g., GKE LoadBalancer does not, while AWS NLB does).
Practical Use Cases
IP‑Based Access Control : Create an Istio
AuthorizationPolicythat denies requests from specific source IP ranges using
remoteIpBlocksor
ipBlocks.
<code>apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: ingress‑policy
namespace: istio‑system
spec:
selector:
matchLabels:
app: istio‑ingressgateway
action: DENY
rules:
- from:
- source:
remoteIpBlocks:
- "2.2.2.2/24"
</code>IP‑Based Load Balancing : Use a
DestinationRulewith
consistentHashon the
x-forwarded-forheader or
useSourceIpto achieve session affinity based on client IP.
<code>apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: example‑destination‑rule
spec:
host: example‑service
trafficPolicy:
loadBalancer:
consistentHash:
httpHeaderName: x-forwarded-for
</code>Summary
Preserving source IP is crucial for access control, load balancing, and analytics.
Envoy uses the
X-Forwarded-Forheader to convey client IP for HTTP traffic.
Setting
externalTrafficPolicy: Localon Services and choosing the appropriate interception mode (REDIRECT or TPROXY) enables correct IP capture for both north‑south and east‑west traffic.
When multiple proxies are involved, configure
numTrustedProxiesto identify the trusted client IP.
For TCP traffic, the Proxy Protocol provides a reliable way to retain the original client IP.
Ops Development Stories
Maintained by a like‑minded team, covering both operations and development. Topics span Linux ops, DevOps toolchain, Kubernetes containerization, monitoring, log collection, network security, and Python or Go development. Team members: Qiao Ke, wanger, Dong Ge, Su Xin, Hua Zai, Zheng Ge, Teacher Xia.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.