Fix Kubernetes Pod DNS Resolution Permanently: CoreDNS Config Over Manual Edits

Why Manual /etc/resolv.conf Edits Don't Persist

A user reported that a Pod could not reach external domains. The Pod's /etc/resolv.conf contained only nameserver 10.96.0.10 (the cluster CoreDNS IP). Manually adding nameserver 114.114.114.114 fixed resolution, but the change disappeared after Pod restart.

The file you see inside a Pod is not from the container image nor from your Dockerfile. It is dynamically generated by kubelet at Pod startup based on three inputs:

Pod's dnsPolicy (DNS policy)

Pod's dnsConfig (custom configuration)

Host's /etc/resolv.conf

Editing the file inside the Pod is temporary; kubelet regenerates it on every restart. Do not manually edit resolv.conf in Pods — it treats symptoms, not the root cause.

Four dnsPolicy Types Explained

Kubernetes provides four DNS policies. Misunderstanding them causes recurring issues:

ClusterFirst (default) : Uses cluster CoreDNS; falls back to host DNS for unresolved names. Suitable for most business Pods.

Default : Inherits host's /etc/resolv.conf directly, bypassing CoreDNS. For Pods that don't need cluster Service access.

ClusterFirstWithHostNet : Uses CoreDNS even when hostNetwork: true. Required for hostNetwork Pods (e.g., Nginx Ingress, monitoring agents).

None : Fully custom via dnsConfig. For special DNS requirements.

80% of DNS issues stem from hostNetwork Pods missing ClusterFirstWithHostNet , causing them to use host DNS and fail to resolve cluster Services.

Three Correct Solutions

Solution 1: Modify CoreDNS ConfigMap (Recommended, Permanent)

Manually adding nameserver 114.114.114.114 in a Pod tells it to query 114 when CoreDNS fails. This logic belongs in CoreDNS, not every Pod.

Edit the CoreDNS ConfigMap: kubectl edit configmap coredns -n kube-system Locate the Corefile. The default forward . /etc/resolv.conf line means: forward unresolved queries to the host's /etc/resolv.conf. Cloud VMs typically use the cloud provider's internal DNS (e.g., Alibaba Cloud 100.100.2.136, Huawei Cloud 100.125.1.250), which usually resolve external domains.

If the cloud DNS is unstable, replace it with public DNS servers: forward . 114.114.114.114 223.5.5.5 Restart CoreDNS to apply:

kubectl delete pod -n kube-system -l k8s-app=kube-dns

One-time configuration, all Pods benefit permanently. New deployments need no DNS changes.

Solution 2: Per-Pod dnsConfig (Workaround)

If you cannot modify CoreDNS, add backup DNS at the Pod level:

spec:  dnsPolicy: ClusterFirst  dnsConfig:    nameservers:    - 114.114.114.114    options:    - name: ndots      value: "2"

The Pod's resolv.conf will then contain both CoreDNS and the custom nameserver: nameserver 10.96.0.10nameserver 114.114.114.114 Pros: Survives Pod restarts. Cons: Must be added to every new Deployment.

Solution 3: Fix hostNetwork Pods with ClusterFirstWithHostNet

For Pods using hostNetwork: true (e.g., Nginx Ingress, monitoring components), explicitly set:

spec:  hostNetwork: true  dnsPolicy: ClusterFirstWithHostNet

Without this, the Pod defaults to host DNS and cannot resolve cluster-internal Services.

Hidden Pitfall: ndots:5 Causes Resolution Timeouts

Kubernetes defaults to options ndots:5 in resolv.conf. This means: if a domain has fewer than 5 dots, the resolver first tries appending each search domain suffix, waiting for each to fail before querying the original name.

Example: requesting redis (1 dot) triggers sequential attempts: redis.default.svc.cluster.local. → fails redis.svc.cluster.local. → fails redis.cluster.local. → fails redis. → finally succeeds

Each attempt waits for DNS timeout (default 5 seconds). Four attempts = 20 seconds latency. Frequent short-name lookups (e.g., redis) kill performance.

Fix: Change to ndots:2 via dnsConfig, or configure a CoreDNS rewrite rule to automatically expand redis to redis.default.svc.cluster.local, skipping the search loop.

Practical Recommendations

Dev/Test environments: Add forward in CoreDNS ConfigMap for a one-time fix.

Production: Point CoreDNS forward to the cloud provider's internal DNS (not 114/223). Internal DNS is optimized for VPC resolution, offering lower latency and better stability; public DNS may be rate-limited.

Quick Troubleshooting Commands

# 1. Check dnsPolicykubectl get pod <pod-name> -o yaml | grep dnsPolicy# 2. View effective resolv.confkubectl exec <pod-name> -- cat /etc/resolv.conf# 3. Test resolution against specific DNS serverskubectl exec <pod-name> -- nslookup www.baidu.com 10.96.0.10kubectl exec <pod-name> -- nslookup www.baidu.com 114.114.114.114

One-Line Summary

When a Pod cannot resolve domains, first check its dnsPolicy. In most cases, adding a forward directive in the CoreDNS ConfigMap solves it permanently. Stop editing files inside Pods — that's a band-aid, not a solution.

Without understanding the mechanism, you can waste a night debugging. With it, it's a one-line config change.

Quick Reference Cheat Sheet

All Pods fail external resolution → Edit CoreDNS ConfigMap, add forward → Permanent ✅

Individual Pod needs backup DNS → Add dnsConfig in Deployment → Not permanent ❌ (repeat per service)

hostNetwork Pod cannot resolve Services → Add dnsPolicy: ClusterFirstWithHostNet → Permanent ✅

Short domain resolution slow (ndots:5) → Change to ndots:2 or CoreDNS rewrite → Permanent ✅