How to Enable Dynamic DNS in Nginx on Kubernetes to Avoid Service Outages

Background

When Nginx is used as a reverse proxy in Kubernetes, the default DNS caching behavior can cause traffic interruption after a backend Pod’s IP address changes.

Root Cause

Nginx resolves the domain name in proxy_pass or upstream only once at start‑up or on nginx -s reload. The resolved IP is cached and never refreshed, so a new Pod IP is ignored, leading to 502 Bad Gateway or timeout errors.

Typical Failure Scenario

A developer configured Nginx to proxy python-www service. After manually deleting the backend Pod, CoreDNS updated the service name, but Nginx kept sending requests to the stale IP, as confirmed by tcpdump.

Solution: Enable Dynamic DNS Resolution

Configure Nginx to query the cluster DNS periodically and to resolve upstream hosts at request time.

1. Set the resolver

In the http block, point resolver to the cluster DNS service (e.g., kube-dns.kube-system.svc.cluster.local) and set a reasonable valid time.

http {
    # Query the Kubernetes DNS service
    resolver kube-dns.kube-system.svc.cluster.local valid=30s;
    # ... other configuration
}

2. Enable resolve on upstream

Add the resolve parameter to the server line inside the upstream block and use a fully‑qualified domain name.

http {
    resolver kube-dns.kube-system.svc.cluster.local valid=30s;

    upstream my_backend {
        # Enable dynamic resolution
        server my-backend-service.default.svc.cluster.local resolve;
    }

    server {
        listen 80;
        location / {
            proxy_pass http://my_backend;
        }
    }
}

If you need to force runtime resolution for proxy_pass, use a variable:

http {
    resolver kube-dns.kube-system.svc.cluster.local valid=30s;

    server {
        listen 80;
        location / {
            set $backend_host my-backend-service.default.svc.cluster.local;
            proxy_pass http://$backend_host:8080;
        }
    }
}

Key Points

Use variables in proxy_pass to avoid IP caching.

Always use FQDN (e.g., my-backend-service.default.svc.cluster.local) because the resolver directive bypasses /etc/resolv.conf search domains.

Final Recommendations

Define resolver in the http block and point it to kube-dns.kube-system.svc.cluster.local.

Set an appropriate valid interval (e.g., 10s‑30s) to balance DNS load and freshness.

In each upstream, add the resolve flag and use the service’s FQDN.

With these settings, Nginx automatically re‑resolves backend addresses after Pod IP changes, eliminating “memory loss” and improving system stability and resilience.