How to Mirror Traffic with Nginx Ingress for Safe Testing and Debugging

Background

Traffic Mirror is used for two scenarios:

During major refactoring or new feature releases, mirror live traffic to an offline environment for simulation testing.

When a performance bottleneck occurs, mirror real traffic to an offline environment for rapid troubleshooting.

Solution Overview

This article explains how to use the Ingress Nginx Controller to duplicate application traffic for system simulation testing and problem localization.

Steps

3.1 Deploy Test Applications

Step 1 Deploy application in Product Cluster

Application load:

Service routing:

Step 2 Deploy the same application in Stage Cluster

Step 3 Obtain application domain names

Product Cluster: www.product-nginx.com

Stage Cluster: www.stage-nginx.com

3.2 Traffic Mirror Configuration

Mirror 100% of traffic from the Product Cluster to the Stage Cluster, i.e., duplicate all requests to www.product-nginx.com and forward them to www.stage-nginx.com. The client receives responses only from the Product Cluster; Stage responses are discarded.

The Stage Cluster Ingress acts solely as a mirror receiver without any configuration changes.

After mirroring, the client sees only the original Product Cluster responses.

Step 1 Configure nginx‑ingress ConfigMap

Add split_clients entries for multiple mirror targets.

kubectl get configmap -n kube-system
kubectl edit configmap nginx-ingress-controller -n kube-system

# Add configuration

data:
  http-snippet: |
    split_clients "$date_gmt" $mirror_servers1 {
      100% www.stage-nginx1.com;
    }
    split_clients "$date_gmt" $mirror_servers2 {
      100% www.stage-nginx2.com;
    }

Configuration notes:

Traffic percentage range is (0, 100] and total must not exceed 100%.

Multiple distinct mirror targets can be configured simultaneously.

Step 2 Configure Ingress in Product Cluster

Use configuration‑snippet and server‑snippet annotations to add mirror directives.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      mirror /mirror1;
      mirror /mirror2;
    nginx.ingress.kubernetes.io/server-snippet: |
      location = /mirror1 {
        internal;
        set $proxy_upstream_name "default-nginx-service-80";
        set $shadow_service_name "nginx-product-service";
        proxy_set_header X-Shadow-Service $shadow_service_name;
        proxy_set_header Host $mirror_servers1;
        proxy_pass http://$mirror_servers1$request_uri;
      }
      location = /mirror2 {
        internal;
        set $proxy_upstream_name "default-nginx-service-80";
        set $shadow_service_name "nginx-product-service";
        proxy_set_header X-Shadow-Service $shadow_service_name;
        proxy_set_header Host $mirror_servers2;
        proxy_pass http://$mirror_servers2$request_uri;
      }
spec:
  rules:
  - host: www.product-nginx.com
    http:
      paths:
      - path: /
        backend:
          service:
            name: nginx-product
            port:
              number: 80
        pathType: ImplementationSpecific

Step 3 Modify CoreDNS hosts configuration

Add domain mappings for the stage services.

kubectl get configmap -n kube-system
kubectl edit configmap coredns -n kube-system

data:
  Corefile: |
    .:5353 {
      bind ${POD_IP}
      hosts {
        192.168.4.16 www.stage-nginx1.com
        192.168.4.53 www.stage-nginx2.com
        fallthrough
      }
    }

Verification

Test the product domain and observe logs:

Product cluster logs show two flows per request, one directed to the stage cluster.

Stage cluster logs confirm receipt of mirrored traffic.

Multiple mirror targets are verified to receive traffic as configured.