Skip to content
Six Dual-Stack IPv6 Migration Traps and How to Verify Each Fix
Operations 34 min read

Six Dual-Stack IPv6 Migration Traps and How to Verify Each Fix

This guide details six common compatibility pitfalls in dual-stack IPv6 deployments—DNS AAAA precedence, socket binding inconsistencies, firewall rule gaps, MTU/PMTUD failures, IPv6 address formatting in proxies and logs, and Kubernetes service/CNI/network policy mismatches—and provides verification commands, configuration examples, and rollback strategies for each.

Raymond Ops
Raymond Ops
Raymond Ops

Define the Dual-Stack Boundary

Dual-stack does not mean simply adding an IPv6 address to every machine. For an external HTTP service, at least five path segments must be verified simultaneously: client→DNS, DNS→ingress, ingress→backend, backend→dependencies, dependency callbacks→service. Any segment that is IPv4-only, or uses a different ACL, routing, NAT, health check, or certificate policy for IPv6, will let "ping works" mask real risk.

Record a baseline before changes. The following script collects state without modifying the system; run on ingress and backend nodes and archive results.

#!/usr/bin/env bash
set -euo pipefail
OUT_DIR="./ipv6-baseline-$(hostname)-$(date +%Y%m%d%H%M%S)"
mkdir -p "${OUT_DIR}"
ip -br address > "${OUT_DIR}/ip-address.txt"
ip -6 route show table all > "${OUT_DIR}/ipv6-routes.txt"
sysctl net.ipv6.conf.all.disable_ipv6 net.ipv6.conf.default.disable_ipv6 > "${OUT_DIR}/ipv6-sysctl.txt"
ss -lntup > "${OUT_DIR}/listeners.txt"
resolvectl status > "${OUT_DIR}/resolver.txt" 2>&1 || true
tar -czf "${OUT_DIR}.tar.gz" "${OUT_DIR}"
ip -br address

confirms which interface holds the address; ip -6 route show table all reveals policy routes or wrong default routes. Do not just save screenshots of ip addr; rollback needs comparable text evidence.

Check that the kernel has not disabled IPv6 globally or per interface:

sysctl -a 2>/dev/null | grep -E '^net\.ipv6\.conf\..*\.disable_ipv6'

Results for all, default, and the actual address-carrying <interface> must all be 0. Note that all=0 cannot auto-fix an interface explicitly set to 1; conversely, a temporary sysctl -w does not replace persistent config.

List local IPv6 addresses and lifetimes. If an address has only a short preferred lifetime, connections may suffer source-address selection anomalies after expiry or renumbering. ip -6 address show dev <interface> Example output should be marked as such, e.g. inet6 2001:db8:100:20::12/64 scope global dynamic. Verify scope global, prefix length, valid/preferred lifetimes, and interface match network design; do not use the documentation prefix 2001:db8::/32 as a real production address.

Trap 1: DNS Publishes AAAA, Clients Prefer an Unavailable Path

Modern clients use Happy Eyeballs with fast fallback, but fallback does not guarantee a seamless experience. Some SDKs, legacy systems, enterprise proxies, or custom resolvers resolve AAAA first and wait for connection timeout; worse, recursive DNS may have cached AAAA while the business only opened half the ingress path.

Query A and AAAA separately; do not rely on commands that only show merged results.

dig +noall +answer A <domain>
dig +noall +answer AAAA <domain>

Run from external, office, and target VPC networks before publishing. DNS console showing the record does not prove the user's recursive server has refreshed. For critical domains, query authoritative DNS:

dig +trace AAAA <domain>
+trace

increases query volume; do not run high-frequency on production probes. Its value is distinguishing missing authoritative records, delegation errors, and recursive cache issues.

Verify HTTP against a fixed ingress address to avoid DNS mixing the problem:

curl -6 --connect-timeout 3 --max-time 10 -sS -D - \
  --resolve <domain>:443:[<ingress-IPv6-address>] \
  "https://<domain>/healthz" -o /dev/null

Brackets are mandatory for IPv6 literals in URL/ --resolve. Response status, certificate SNI, and Host must match IPv4. Change -6 to -4 for a same-condition control; compare not only connectivity but also 30x redirects, auth, and rate-limiting results.

A canary-friendly check script (exits non-zero on any failure, suitable for pipelines):

#!/usr/bin/env bash
set -euo pipefail
DOMAIN="<domain>"
PATH_NAME="/healthz"
for family in 4 6; do
  echo "checking IPv${family}"
  curl "-${family}" --fail --silent --show-error \
    --connect-timeout 3 --max-time 10 \
    "https://${DOMAIN}${PATH_NAME}" >/dev/null
done
echo "dual-stack endpoint check passed"

Canary order: deploy ingress IPv6 and monitoring first, verify with internal test domain, then publish AAAA with short TTL to a small-traffic subdomain, observe a full DNS cache cycle, finally merge into main domain. Rollback is not deleting server IPv6 but revoking AAAA or pointing it to a verified ingress; confirm no IPv6-only internal callers before revoking.

Trap 2: Listening Sockets and Address-Family Policy Mismatch

Whether a service listening on [::]:443 also accepts IPv4 connections depends on net.ipv6.bindv6only and the application's own bind behavior. Do not extrapolate from one machine to all nodes. Nginx's listen [::]:443 does not replace listen 443; production config should declare both explicitly and keep TLS, HTTP/2, logging, and access-control parameters identical.

View actual listeners and address families: ss -H -lntp '( sport = :80 or sport = :443 )' If only *:443 appears, combine ss -4 and ss -6 to confirm. Read kernel policy: sysctl net.ipv6.bindv6only Recommended explicit Nginx listen config (avoid relying on IPv4-mapped IPv6 behavior):

server {
  listen 80;
  listen [::]:80;
  server_name <domain>;
  return 308 https://$host$request_uri;
}

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name <domain>;
  ssl_certificate      /etc/nginx/tls/<domain>.crt;
  ssl_certificate_key  /etc/nginx/tls/<domain>.key;
  location = /healthz { return 200 "ok
"; }
}

Backup before changes, then syntax-check and reload gracefully.

sudo cp -a /etc/nginx/nginx.conf "/etc/nginx/nginx.conf.bak.$(date +%Y%m%d%H%M%S)"
sudo nginx -t
sudo systemctl reload nginx
sudo ss -lntp '( sport = :80 or sport = :443 )'

If syntax check fails, stop immediately; do not restart. Rollback using the backup and re-run nginx -t then reload.

For containerized services, an app binding only 127.0.0.1 makes host IPv6 ingress unreachable. Check inside the container's network namespace, not just host port mappings:

docker exec <container> sh -c 'ss -lntp || netstat -lntp'

Docker IPv6 support depends on daemon config and network creation. Example dedicated bridge network (subnet must be allocated by network team, not picked from public prefix):

services:
  api:
    image: <image>:<tag>
    networks: [dualstack]
networks:
  dualstack:
    enable_ipv6: true
    ipam:
      config:
        - subnet: "<IPv6-prefix>/64"

Changing Docker daemon or default network affects all containers on the node; must canary on isolated node first. Do not restart production Docker daemon to fix a single service.

Trap 3: Firewall and Security Groups Only Allow IPv4

IPv4 and IPv6 are two independent filter rule sets. Common misjudgment: ufw status shows port allowed, but config file has IPV6=no; or cloud security group only has CIDR 0.0.0.0/0. IPv6 lacks NAT as a default "shield"; routable addresses mean ingress rules must be more precise.

Collect host-side rules first; only the rule system actually enabled on the host counts. For nftables systems: sudo nft list ruleset In iptables-nft compat environments, check v4 and v6 separately; iptables -S does not represent IPv6:

sudo iptables -S
sudo ip6tables -S

With UFW, confirm IPv6 switch and actual rules:

grep -E '^IPV6=' /etc/default/ufw
sudo ufw status numbered

For internet ingress, allow only necessary protocols and reviewed source prefixes. Example nftables snippet allowing IPv6 TCP 443; merge into existing inet filter table structure to avoid overwriting other chains.

table inet filter {
  chain input {
    type filter hook input priority filter; policy drop;
    ct state established,related accept
    iifname "lo" accept
    ip6 nexthdr ipv6-icmp accept
    tcp dport 443 ip6 saddr <allowed-IPv6-source-prefix> accept
  }
}

Do not blindly block all ICMPv6. Neighbor discovery, path MTU discovery, and error reporting depend on it; over-restriction often shows as "small packets work, large responses stall". Final rules must follow security baseline, at least retaining network-team-approved necessary types.

Firewall changes are high-risk: keep an out-of-band console session, use timed rollback to protect remote access. The script below checks candidate rules with nft -c and restores backup after 5 minutes; only cancel the at-job after verification.

#!/usr/bin/env bash
set -euo pipefail
BACKUP="/root/nftables.before-ipv6.$(date +%Y%m%d%H%M%S)"
sudo nft list ruleset | sudo tee "${BACKUP}" >/dev/null
sudo nft -c -f /etc/nftables.conf
echo "sudo nft -f '${BACKUP}'" | at now + 5 minutes
sudo nft -f /etc/nftables.conf
echo "After verification, use atq/atrm to cancel auto-rollback job"

Ensure atd is installed and enabled before running; if not, do not assume the protection exists. Verify from authorized external probe with IPv6 TLS connection, then check counters or service logs. Cloud security group, load balancer ACL, and host firewall—all three must be consistent.

Trap 4: MTU, Path MTU Discovery, and Fragmentation Ignored

IPv6 routers do not fragment; when PMTUD is blocked by firewalls, typical symptom: three-way handshake succeeds, short responses succeed, but larger packets (downloads or TLS handshake) time out. Do not just lower interface MTU arbitrarily; first locate actual path and ICMPv6 reachability.

Check interface MTU and default route:

ip link show dev <interface>
ip -6 route get <peer-IPv6-address>

Linux tracepath6 observes probed path MTU; target can be an authorized backend or ingress: tracepath6 -n <peer-IPv6-address> Peer not responding to probes does not directly prove MTU issue; correlate with service failure times, packet captures, and ICMPv6 Packet Too Big evidence. Capture with care (avoid long runs on interfaces carrying sensitive data):

sudo timeout 30 tcpdump -ni <interface> -vv 'icmp6 or (tcp port 443 and ip6)'

Example output must be labeled as such: ICMP6, packet too big, mtu 1280 indicates a device signaling a smaller usable MTU; check intermediate firewalls forward that ICMPv6, and account for tunnel/VPN/overlay encapsulation overhead.

If service sits behind Nginx, TCP proxy MSS clamping should be done by network team at border devices. Application layer must not replace network fix with hard-coded response chunking. HTTP layer can verify transport with a large but non-sensitive test object:

curl -6 --fail --http1.1 --max-time 30 \
  -o /dev/null -w 'http=%{http_code} bytes=%{size_download} time=%{time_total}
' \
  "https://<domain>/ipv6-test-1m.bin"

If temporarily lowering a single interface MTU, confirm interface scope and persistent source after reboot. Command below only for controlled validation within change window; disconnection risk must be communicated to business beforehand:

sudo ip link set dev <interface> mtu <temporary-MTU-value>
ip link show dev <interface>

Rollback to pre-change MTU. Do not treat 1280 as a universal optimum; it is the IPv6 minimum link MTU, performance and encapsulation scenarios need real measurement.

Trap 5: Proxies, Upstreams, and Logs Mishandle IPv6 Address Format

IPv6 literal addresses contain colons. Concatenating $remote_addr:$remote_port directly into log fields, or passing IPv6 to a legacy parser expecting host:port, breaks audit parsing, whitelist matching, and can misroute requests. Correct approach: use structured fields, or brackets in URIs.

Check whether Nginx actually passes the address to upstream, not just access log:

log_format json_access escape=json
'{"time":"$time_iso8601","remote_addr":"$remote_addr",'
'"request":"$request","status":$status,"upstream":"$upstream_addr"}';
access_log /var/log/nginx/access.json json_access;
escape=json

prevents request field from breaking JSON; log collector must parse as JSON, not space-split. After change, run nginx -t, then issue one IPv4 and one IPv6 request and inspect fields.

Proxying to IPv6 literal upstream requires brackets, otherwise Nginx cannot distinguish port:

upstream api_backend {
  server [<backend-IPv6-address>]:8080 max_fails=3 fail_timeout=10s;
}
server {
  listen 443 ssl;
  listen [::]:443 ssl;
  location /api/ { proxy_pass http://api_backend; }
}

If using domain upstream and wanting Nginx to resolve AAAA at runtime, specify an IPv6-capable resolver and explicitly test DNS change behavior. Different Nginx versions handle dynamic resolution differently; refer to local nginx -V and deployment spec, not blindly add variables in production config.

For real client IP, only trust X-Forwarded-For or PROXY protocol when load balancer address is strictly trusted. Config below restricts trusted proxies to explicit IPv4/IPv6 CIDRs:

set_real_ip_from <load-balancer-IPv4-CIDR>;
set_real_ip_from <load-balancer-IPv6-prefix>;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

Widening set_real_ip_from to 0.0.0.0/0 or ::/0 lets any client forge source address; never do that. Verify by comparing ingress connection address, forwarded header, and $realip_remote_addr in backend logs to confirm the chain.

Application database, cache, or SMTP callback configs often have the same issue. URIs must bracket IPv6, e.g.:

postgresql://<user>:<pass>@[<db-IPv6-address>]:5432/<dbname>

This is a config example; do not write passwords into command history, images, or Git repos. Prefer secret management or restricted-permission env files.

Trap 6: Kubernetes Service, CNI, NetworkPolicy, and Monitoring Not Dual-Stack Consistent

Kubernetes dual-stack requires control plane, nodes, CNI, Pod CIDR, Service CIDR, and kube-proxy/data plane to all support both families. Just adding an IPv6 address to Ingress does not mean Pod-to-Pod, Pod-to-internet, or Service DNS have IPv6. Confirm cluster version and address-family capabilities first; do not edit kubeadm or CNI production config before verification.

kubectl version --short
kubectl get nodes -o wide
kubectl -n <namespace> get pods -o wide

Check specific Service's ipFamilies, ipFamilyPolicy, and clusterIP list:

kubectl -n <namespace> get service <service-name> -o yaml

Dual-stack Service example manifest. PreferDualStack allows degradation when cluster capability is incomplete, suitable for migration; only evaluate RequireDualStack when dual-stack is truly required (it may prevent Service creation in unsupported environments).

apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: <namespace>
spec:
  selector:
    app: api
  ports:
    - name: http
      port: 80
      targetPort: 8080
  ipFamilyPolicy: PreferDualStack
  ipFamilies:
    - IPv4
    - IPv6

Application side should resolve and connect to Service from inside a Pod, avoiding node-network success masking CNI issues:

kubectl -n <namespace> exec <test-pod> -- sh -c '
  getent ahostsv4 <service-name> && getent ahostsv6 <service-name>
  wget -4 -qO- http://<service-name>/healthz
  wget -6 -qO- http://<service-name>/healthz
'

Not all minimal images have wget -4/-6; run in a dedicated diagnostic image or use tools actually present. Do not install software temporarily in business containers and leave state.

NetworkPolicy must also check IPBlock address family. Snippet below shows IPv6 source restriction syntax; merge after auditing existing ingress rules; empty ingress or wrong selector may accidentally cut all traffic.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-ingress-ipv6
  namespace: <namespace>
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes: [Ingress]
  ingress:
    - from:
        - ipBlock:
            cidr: <allowed-IPv6-prefix>
      ports:
        - protocol: TCP
          port: 8080

Before applying NetworkPolicy, use a test namespace and canary Pod, record existing policies:

kubectl -n <namespace> get networkpolicy -o yaml > networkpolicy.before.yaml
kubectl -n <namespace> apply --dry-run=server -f api-ingress-ipv6.yaml
kubectl -n <namespace> apply -f api-ingress-ipv6.yaml
--dry-run=server

only validates API acceptance, not that CNI will enforce as expected. After apply, probe from allowed and disallowed sources; if business impacted, restore from exported manifest and re-verify.

kubectl -n <namespace> apply -f networkpolicy.before.yaml
kubectl -n <namespace> get networkpolicy

Monitoring must observe success rate and latency per address family. Metric names depend on Ingress, Service Mesh, or exporter; below PromQL is illustrative for common HTTP request counter—must match actual exporter metrics and labels:

sum by (ip_family, code) (
  rate(http_requests_total{service="<service-name>"}[5m])
)

If system lacks ip_family label, do not assume it exists in dashboards; add a safe protocol-family field from ingress logs, then aggregate via log platform. Node-level can collect node_network_receive_bytes_total etc., but again must match actual node-exporter metrics.

Go-Live Criteria, Alerting, and Rollback Sequence

An acceptable dual-stack launch is not "no obvious IPv6 alerts" but targeted evidence proving both paths work: DNS A/AAAA match design; ingress listens on v4/v6; security groups, LB ACLs, host firewalls consistent; app health checks, auth, upload/download, callbacks all pass; per-address-family error rate and latency stable during observation window.

Post-deploy patrol script (runs as cron; verifies resolution, IPv4, IPv6, Nginx config; does not replace real external user probes):

#!/usr/bin/env bash
set -euo pipefail
DOMAIN="<domain>"
HEALTH_PATH="/healthz"
test -n "$(dig +short A ${DOMAIN})"
test -n "$(dig +short AAAA ${DOMAIN})"
for family in 4 6; do
  curl "-${family}" -fsS --connect-timeout 3 --max-time 10 \
    "https://${DOMAIN}${HEALTH_PATH}" >/dev/null
done
nginx -t
echo "$(date -Is) dual-stack check ok"

Rollback by failure layer: if only public IPv6 fails, withdraw AAAA from traffic ingress or restore previous record set in DNS console; if ingress config wrong, restore verified Nginx config; if NetworkPolicy blocks, restore exported policy; if routing or MTU fails, revert network changes and keep packet captures, routes, and error timeline. After any rollback, verify IPv4 unaffected, then explicitly state IPv6 status—avoid reporting "temporarily IPv4-only" as "full recovery".

After migration, treat address family as a first-class dimension in capacity, availability, and security audits. IPv6 must not become an unmaintained side path: change review, drills, log parsing, alerting, WAF/ACL, and dependency onboarding specs all enter daily ops.

Change Record Evidence Checklist

Dual-stack change ticket must include: address planning, all exposed ports, DNS record sets, ingress and backend v4/v6 connectivity results, ACL diffs, certificate validation results, observability dashboard links, rollback owner. Especially callback dependencies: payments, object storage, OAuth, webhooks, SMTP, message queues, third-party SaaS. They may resolve AAAA but source, proxy, or allowlist still only allow IPv4.

For server certificates, check SNI and cert chain on both protocol families; this can reveal an IPv6 ingress pointing to an old TLS termination node.

for family in 4 6; do
  echo "IPv${family}"
  openssl s_client -"${family}" -connect <domain>:443 -servername <domain> \
    -verify_return_error </dev/null 2>/dev/null | \
    openssl x509 -noout -subject -issuer -dates
done

Local openssl may not support -4/-6; if help lacks those, resolve addresses first via getent or dig, then use bracketed IPv6 literal with -connect. Commands must not output or save private keys.

During release window, alert on IPv6 errors independently. Below query illustrates metric formed from parsed Nginx logs; metric names and labels must match actual log collector and exporter.

sum(rate(nginx_http_requests_total{server="<domain>",ip_family="ipv6",status=~"5.."}[5m]))
/
sum(rate(nginx_http_requests_total{server="<domain>",ip_family="ipv6"}[5m]))

If no protocol-family label, do not use empty result to claim IPv6 has no errors. First confirm log field generation, collection rules, and dashboard label cardinality; then sample-verify via access log remote address or ingress socket.

For outbound dependencies, application's IPv4/IPv6 selection behavior must be pinned. Do not rely on DNS return order; client libraries, OS RFC 6724 policy, proxy env vars, and connection pool caches can all change actual choice. Modifying gai.conf, system-wide priority, or disabling address family has wide impact; should be a separate network change approval, not a last-minute tweak during business launch.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

IPv6kubernetesfirewallNginxDNSMTUdual-stackNetworkPolicy
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.