Practical Guide to Kubernetes Service Discovery: Theory to Production-Ready Design

Why Service Discovery Determines System Limits

In Kubernetes, service discovery is not just name resolution; it is a runtime infrastructure that spans deployment, routing, traffic governance, elastic scaling, fault removal, and multi‑cluster connectivity.

Typical request flow:

Client / Browser
  -> CDN / WAF / Global DNS
  -> LoadBalancer
  -> Ingress / Gateway
  -> Service
  -> kube-proxy / eBPF dataplane
  -> Pod
  -> Downstream Service
  -> CoreDNS / EndpointSlice / Mesh xDS

Production issues often arise in gray areas such as pods receiving traffic before they are Ready, stale connections after scaling, layered DNS caches, slow kube‑proxy sync, hotspot pods under high concurrency, and cross‑region routing failures.

Availability – can failed instances be removed quickly?

Performance – is resolution, forwarding, and connection reuse stable?

Resilience – does scaling affect traffic within seconds?

Observability – can you pinpoint whether DNS, routing, connection pool, or the application is at fault?

Architectural extensibility – can you smoothly upgrade to Ingress, Mesh, Gateway API, or multi‑cluster?

Building a Complete Service‑Discovery Architecture Map

2.1 Control Plane – "Who Can Receive Traffic"

Service

: declares abstract service and entry point. EndpointSlice Controller: generates backend lists based on selectors and pod readiness. CoreDNS: resolves service names to ClusterIP or headless records. kube-proxy: compiles Service to node‑local forwarding rules. Ingress / Gateway Controller: handles L7 entry and routing rules. Service Mesh Control Plane: distributes xDS configs when stronger governance is needed.

2.2 Data Plane – Actual Traffic Handling

iptables

/ IPVS / eBPF: L4 traffic forwarding. Envoy / Nginx / HAProxy: L7 proxy and traffic governance.

Application connection pools for HTTP/gRPC/DB long‑connections.

Local DNS caches in OS, JVM, language runtimes.

2.3 Name Layer – Service, Domain, Instance Names

payment

– Service name. payment.prod.svc.cluster.local – Fully qualified domain name. payment-0.payment.prod.svc.cluster.local – Headless single‑instance name.

These map to different discovery semantics: stable entry, real instance set, or mesh‑registered workload.

2.4 Governance Layer – Post‑Discovery Capabilities

Circuit breaking, rate limiting, timeout, retry.

Canary, blue‑green, header‑based gray releases.

Cross‑AZ/Region proximity routing.

mTLS, authentication, authorization.

Tracing, metrics, anomaly attribution.

Service Fundamentals

3.1 Core Problem Service Solves

Pods are volatile – they get new IPs on recreation, scale up/down, and coexist during upgrades. Callers must rely on a stable abstraction, which Service provides:

Stable virtual address ( ClusterIP).

Stable DNS name.

Dynamic binding to the backend pod list.

3.2 Request Flow Through a Service

order Pod
  -> resolve payment.prod.svc.cluster.local
  -> CoreDNS returns payment ClusterIP
  -> request sent to ClusterIP:Port
  -> kube-proxy/eBPF matches forwarding rule
  -> selects a Ready payment Pod
  -> DNAT / proxy forwards to real PodIP:TargetPort

DNS usually resolves only to the ClusterIP, not directly to pods.

The data‑plane forwarding rule decides the final pod. readinessProbe success directly influences pod inclusion in the backend set.

3.3 Production‑Critical Service Fields

selector

: mis‑labeling yields empty Endpoints. targetPort: mismatched container ports cause connectivity failures. sessionAffinity: enabling can create hotspots. publishNotReadyAddresses: only for StatefulSet, registries, or init components.

3.4 Service Limits

Service knows only pod readiness, not business‑level health.

It cannot sense thread‑pool saturation.

It cannot automatically handle timeouts, retry storms, or cascade failures.

Therefore readinessProbe is more aligned with business availability than livenessProbe, and applications must implement their own timeout, retry, circuit‑break, and graceful shutdown logic.

EndpointSlice – The Real Backend Source

4.1 Why EndpointSlice Beats Endpoints

Single large object updates are costly.

With 1000+ backends the object size explodes.

API server, watcher, and kube‑proxy pressure increase.

EndpointSlice splits a Service's backend into multiple slices, each with limited capacity, allowing incremental updates.

4.2 Generation Logic

Service selector matches Pods.

Pod readiness state.

Pod node, zone, address type.

Port, protocol, topology info.

Typical inspection commands:

kubectl get endpointslice -n prod -l kubernetes.io/service-name=payment
kubectl describe endpointslice -n prod payment-xxxxx

4.3 ReadinessProbe Impact

Pods failing readiness are excluded from standard Service backends.

kube‑proxy will not forward new requests to them.

Ingress, Gateway, and Mesh also build backend pools from this view.

Design rules for readiness probes:

Check must reflect "Can I accept business traffic now?"

Check must be fast, stable, and side‑effect‑free.

A typical anti‑pattern is putting heavy DB checks into readiness, causing batch removal of instances during short spikes.

4.4 Recommended Probe Split

livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 3

startupProbe:
  httpGet:
    path: /actuator/health/startup
    port: 8080
  failureThreshold: 30
  periodSeconds: 5

startupProbe

– cold‑start completion. livenessProbe – process liveness. readinessProbe – ability to accept business traffic.

CoreDNS Deep Dive

5.1 Why Pods Can Resolve Service Names Directly

nameserver 10.96.0.10
search prod.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

In namespace prod, a request for payment triggers the resolver to try:

payment.prod.svc.cluster.local

payment.svc.cluster.local

payment.cluster.local

5.2 CoreDNS Processing Chain

Pod DNS query
  -> CoreDNS Service
  -> CoreDNS Pod
  -> kubernetes plugin checks local watch cache
  -> hits Service / EndpointSlice info
  -> returns ClusterIP or headless instance list

CoreDNS rarely queries the API server; it relies on a local watch cache.

Performance bottlenecks are usually cache hit‑rate and cache‑expiry jitter, not the raw lookup.

5.3 Standard CoreDNS Config and Plugin Roles

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health { lameduck 5s }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf { max_concurrent 1000 }
        cache 30
        loop
        reload
        loadbalance
    }

kubernetes

– resolves in‑cluster services. cache – caches DNS results; determines latency and stability under high QPS. forward – forwards external queries. loadbalance – shuffles multiple records. health/ready – aids probes and traffic removal. prometheus – exposes metrics.

5.4 DNS Cache – A Double‑Edged Sword

Production failures are often "stale" values rather than "not found". Cache layers include CoreDNS, NodeLocal DNSCache, OS cache, language runtime cache, and connection‑pool DNS entries. JVM, for example, can cache DNS for a long time unless networkaddress.cache.ttl is set (e.g., 30 s) and networkaddress.cache.negative.ttl (e.g., 5 s).

5.5 High‑Concurrency Recommendation – Enable NodeLocal DNSCache

For large clusters or high DNS QPS, enable NodeLocal DNSCache. Benefits:

Reduces CoreDNS cross‑node traffic.

Mitigates UDP packet loss tail latency.

Improves cache hit rate.

Pushes DNS queries down to the node.

Suitable scenarios: many Services, short‑lived or high‑frequency DNS queries, high pod density, elevated CoreDNS P99 latency.

5.6 Production‑Grade CoreDNS Tuning

Run at least 2‑3 replicas; scale to 3‑5 based on cluster size.

Add a PodDisruptionBudget to avoid full outage during upgrades.

Use topologySpreadConstraints for node distribution.

Enable Prometheus metrics: dns_request_count_total, latency, cache hit rate.

Enable NodeLocal DNSCache for large clusters.

Keep TTL modest (5‑30 s for service records).

kube‑proxy, iptables, IPVS & eBPF – Forwarding Efficiency

6.1 kube‑proxy Is Not a Proxy

Watches Service / EndpointSlice changes.

Compiles them into node‑local forwarding rules.

Actual forwarding is done by the kernel network stack or eBPF programs.

6.2 iptables Mode

Uses NAT rule chains to match Service IP/port.

On match, DNATs to a backend Pod.

Pros: best compatibility, mature stability.

Cons: high rule count leads to maintenance cost; sync and lookup efficiency degrade with massive Service/Endpoint changes.

6.3 IPVS Mode

Leverages kernel‑mode LVS/IPVS for virtual services and real servers.

Supports multiple scheduling algorithms.

Pros: better for high concurrency and large scale, higher update/search efficiency, friendlier to long‑lived connections.

Typical scheduling strategies: rr (round‑robin), wrr (weighted round‑robin), lc (least‑connection), sh (source‑hash).

Recommendation: most production clusters prefer IPVS; consider eBPF (Cilium) for even larger scale.

6.4 eBPF Data Plane

Service forwarding logic inlined into the kernel via eBPF (e.g., Cilium).

Reduces iptables rule bloat.

Provides richer observability and topology‑aware performance.

Suitable for large multi‑tenant clusters, high observability requirements, and unified Service/NetworkPolicy capabilities.

6.5 conntrack Bottlenecks in High‑Concurrency

Massive short‑lived connections, NAT, and retry storms can exhaust nf_conntrack:

Table full → new connections fail.

Latency jitter spikes.

Accumulation of SYN_SENT and TIME_WAIT.

sysctl -w net.netfilter.nf_conntrack_max=1048576
sysctl -w net.ipv4.ip_local_port_range="10240 65535"
sysctl -w net.ipv4.tcp_tw_reuse=1

Combine with HTTP/gRPC connection reuse, reduced blind retries, client‑side back‑off, and proper timeout settings at gateways and downstream services.

Service Type Selection – Layered by Traffic Entry

7.1 ClusterIP – Default for Internal Communication

Use cases: microservice calls, monitoring, middleware access.

Characteristics: cluster‑internal only, lowest cost, default for most services.

apiVersion: v1
kind: Service
metadata:
  name: user-service
  namespace: prod
spec:
  type: ClusterIP
  selector:
    app: user-service
  ports:
    - name: http
      port: 80
      targetPort: 8080

7.2 Headless Service – Direct Instance Discovery

Use cases: StatefulSet, master‑slave, sharding, client‑side load balancing.

apiVersion: v1
kind: Service
metadata:
  name: redis-cluster
  namespace: prod
spec:
  clusterIP: None
  selector:
    app: redis
  ports:
    - name: redis
      port: 6379
      targetPort: 6379

Returns a list of pod IPs instead of a single ClusterIP.

7.3 NodePort – Development/Debug, Not Production

Suitable for temporary debugging, local integration.

Unsuitable for large‑scale production, unified traffic governance, or strict security.

7.4 LoadBalancer – Cloud‑Native External Entry

Suitable for exposing TCP/UDP services directly.

Acts as the entry point for gateways, Ingress controllers, or mesh gateways.

Best practice: let LoadBalancer handle external traffic entry; delegate complex L7 routing to Ingress or Gateway API.

7.5 ExternalName – Migration Bridge, Not a General Proxy

Essentially a DNS CNAME to an external domain.

Does not participate in kube‑proxy forwarding; cannot apply L4 traffic control.

Ingress, Gateway API & Seven‑Layer Service Discovery

8.1 Why Service Alone Is Insufficient

Service operates at L4. Modern systems also need domain‑based routing, path routing, TLS termination, header/query routing, WAF, authentication, and rate limiting – capabilities provided by Ingress or Gateway API.

8.2 Ingress Positioning

Ingress = L7 traffic rules + concrete controller implementation. Common stack:

Internet -> Cloud LoadBalancer -> Nginx Ingress Controller -> Ingress Rule -> Service -> Pod

8.3 Production‑Ready Nginx Ingress Example

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ecommerce
  namespace: prod
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-body-size: "20m"
    nginx.ingress.kubernetes.io/limit-rps: "200"
    nginx.ingress.kubernetes.io/enable-access-log: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api.example.com
      secretName: api-example-com-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /orders
            pathType: Prefix
            backend:
              service:
                name: order-service
                port:
                  number: 80
          - path: /payments
            pathType: Prefix
            backend:
              service:
                name: payment-service
                port:
                  number: 80

8.4 Gateway API – The Evolution of Ingress

Clearer resource model and role separation.

Stronger routing capabilities: HTTPRoute, GRPCRoute, TCPRoute.

Better fit for platform‑team vs. business‑team collaboration.

Typical relationship: platform team maintains GatewayClass and Gateway; business team maintains HTTPRoute.

Service Mesh in Service Discovery

9.1 When Mesh Takes Over Discovery Duties

When microservice count exceeds 30‑50, multiple languages coexist, gray releases, circuit breaking, and fine‑grained observability are required, native Service becomes insufficient.

Version‑tagged address sets.

Policy‑driven subset selection.

Topology‑aware routing.

Real‑time outlier removal.

9.2 Mesh Discovery Flow (Istio Example)

Pilot / istiod
  -> watches Service / EndpointSlice / Pod
  -> generates xDS configs
  -> pushes to Envoy sidecar / ambient dataplane
  -> Envoy routes via CDS/EDS/RDS/LDS

Kubernetes objects remain the data source; the mesh control plane translates them into a governed service view, while the data‑plane proxies enforce policies.

9.3 Production Canary Release Example

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payment-dr
  namespace: prod
spec:
  host: payment-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 500
      http:
        http1MaxPendingRequests: 1000
        maxRequestsPerConnection: 100
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
    subsets:
      - name: stable
        labels:
          version: v1
      - name: canary
        labels:
          version: v2
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payment-vs
  namespace: prod
spec:
  hosts:
    - payment-service
  http:
    - match:
        - headers:
            x-canary:
              exact: "true"
      route:
        - destination:
            host: payment-service
            subset: canary
    - route:
        - destination:
            host: payment-service
            subset: stable
          weight: 95
        - destination:
            host: payment-service
            subset: canary
          weight: 5
      timeout: 2s
      retries:
        attempts: 2
        perTryTimeout: 800ms
        retryOn: gateway-error,connect-failure,refused-stream,5xx

Application‑Side Practices – Service Discovery Is a Two‑Way Contract

10.1 Java/Spring Boot HTTP Client Example

package com.example.order.client;

import io.netty.channel.ChannelOption;
import io.netty.handler.timeout.ReadTimeoutHandler;
import io.netty.handler.timeout.WriteTimeoutHandler;
import java.time.Duration;
import java.util.concurrent.TimeUnit;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.web.reactive.function.client.ExchangeStrategies;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.netty.http.client.HttpClient;
import reactor.netty.resources.ConnectionProvider;

@Configuration
public class PaymentClientConfig {

    @Bean
    public WebClient paymentWebClient() {
        ConnectionProvider provider = ConnectionProvider.builder("payment-http-pool")
                .maxConnections(500)
                .pendingAcquireMaxCount(2000)
                .maxIdleTime(Duration.ofSeconds(30))
                .maxLifeTime(Duration.ofMinutes(5))
                .evictInBackground(Duration.ofSeconds(60))
                .build();

        HttpClient httpClient = HttpClient.create(provider)
                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 1000)
                .responseTimeout(Duration.ofSeconds(2))
                .doOnConnected(conn -> conn
                        .addHandlerLast(new ReadTimeoutHandler(2, TimeUnit.SECONDS))
                        .addHandlerLast(new WriteTimeoutHandler(2, TimeUnit.SECONDS)));

        return WebClient.builder()
                .baseUrl("http://payment-service.prod.svc.cluster.local")
                .defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
                .clientConnector(new ReactorClientHttpConnector(httpClient))
                .exchangeStrategies(ExchangeStrategies.builder()
                        .codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(2 * 1024 * 1024))
                        .build())
                .build();
    }
}

Recommended JVM flags for DNS TTL control:

-Dsun.net.inetaddr.ttl=30
-Dsun.net.inetaddr.negative.ttl=5

10.2 gRPC Client Recommendations

ManagedChannel channel = NettyChannelBuilder.forTarget("dns:///payment-grpc.prod.svc.cluster.local:50051")
        .defaultLoadBalancingPolicy("round_robin")
        .keepAliveTime(30, TimeUnit.SECONDS)
        .keepAliveTimeout(10, TimeUnit.SECONDS)
        .maxRetryAttempts(3)
        .usePlaintext()
        .build();

During graceful shutdown, servers should fail readiness first, then wait for connections to drain.

Clients must enable reconnection, keepalive, and per‑request deadlines.

10.3 Go HTTP Client Production Example

package client

import (
    "context"
    "net"
    "net/http"
    "time"
)

func NewPaymentHTTPClient() *http.Client {
    dialer := &net.Dialer{Timeout: 1 * time.Second, KeepAlive: 30 * time.Second}
    transport := &http.Transport{
        Proxy: http.ProxyFromEnvironment,
        DialContext: dialer.DialContext,
        ForceAttemptHTTP2: true,
        MaxIdleConns: 500,
        MaxIdleConnsPerHost: 100,
        MaxConnsPerHost: 200,
        IdleConnTimeout: 90 * time.Second,
        TLSHandshakeTimeout: 2 * time.Second,
        ResponseHeaderTimeout: 2 * time.Second,
        ExpectContinueTimeout: 1 * time.Second,
    }
    return &http.Client{Timeout: 3 * time.Second, Transport: transport}
}

func CallPayment(ctx context.Context, client *http.Client) (*http.Response, error) {
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://payment-service.prod.svc.cluster.local/api/payments/health", nil)
    if err != nil { return nil, err }
    return client.Do(req)
}

10.4 Graceful Drain for Pods

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: prod
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: app
          image: example/payment-service:1.2.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 1
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 1
          startupProbe:
            httpGet:
              path: /actuator/health/startup
              port: 8080
            periodSeconds: 5
            failureThreshold: 24
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - |
                    wget -qO- http://127.0.0.1:8080/actuator/drain || true
                    sleep 20

Drain steps:

preStop switches application to drain state.

Readiness immediately fails, stopping new traffic.

Existing connections are allowed to finish.

Process exits after graceful period.

Production Case Study – E‑Commerce Transaction System

11.1 Business Background

Core services: gateway, user‑service, order‑service, payment‑service, inventory‑service, promotion‑service, risk‑service. Goals: 80k QPS peak, P99 < 200 ms, gray releases, single‑AZ fault tolerance.

11.2 Recommended Architecture

User traffic
  -> Global DNS / CDN
  -> Cloud LoadBalancer
  -> Gateway API / Nginx Ingress
  -> gateway‑service
  -> internal ClusterIP Service
  -> business services (order, payment, …)

Control plane
  -> Service
  -> EndpointSlice
  -> CoreDNS
  -> kube‑proxy(IPVS) / Cilium eBPF

Governance
  -> HPA
  -> PDB
  -> Pod Anti‑Affinity
  -> Service Mesh (as needed)

11.3 Key Design Decisions

External entry: LoadBalancer + Gateway/Ingress.

Internal services default to ClusterIP.

Stateful components (Redis, Kafka, MySQL) use Headless Service + StatefulSet.

Prefer IPVS for kube‑proxy; evaluate Cilium for larger scale.

CoreDNS at least 3 replicas with monitoring.

High DNS QPS clusters enable NodeLocal DNSCache.

Uniform application timeout, connection pool, retry, and circuit‑break standards.

11.4 Why Not Direct Headless for All Services

Clients must handle instance list changes.

SDK behaviours differ across languages.

Retry and load‑balancing become inconsistent.

Operational overhead rises sharply.

Principle: stateless microservices use standard Service; only components needing stable identity use Headless.

11.5 High‑Concurrency Scaling Loop

HPA adds Pods.

New Pods start. startupProbe passes. readinessProbe passes.

EndpointSlice updates.

kube‑proxy syncs.

New traffic reaches pods.

Clients establish new connections.

Any slowdown in this chain erodes scaling benefits. Recommendations: optimise cold‑start, pre‑warm hot services, monitor connection‑establishment latency for long‑lived connections.

Multi‑Cluster & Multi‑Region Service Discovery

12.1 Core Challenges

Which cluster to call?

Proximity routing.

Failover when primary cluster is down.

Latency and consistency across clusters.

12.2 Common Solutions

Solution 1 – Global DNS + Independent Cluster Discovery

Simple, region‑autonomous, low cross‑cluster governance needs.

Solution 2 – Multi‑Cluster Service Mesh

Unified authentication, observability, cross‑cluster routing, failover, traffic mirroring.

Solution 3 – Registry Bridge (Consul, Nacos, Eureka)

Useful during migration or mixed VM/container environments.

12.3 Design Principles

Avoid making all traffic cross‑cluster by default.

Prefer local autonomy; use cross‑cluster as fallback.

Design discovery together with data replication.

Define clear downgrade paths for both traffic and data.

Five‑Layer Troubleshooting Methodology

13.1 Name Resolution Layer

kubectl exec -it deploy/order-service -n prod -- nslookup payment-service
kubectl exec -it deploy/order-service -n prod -- cat /etc/resolv.conf
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=200

13.2 Backend Generation Layer

kubectl get svc payment-service -n prod -o wide
kubectl get endpointslice -n prod -l kubernetes.io/service-name=payment-service
kubectl describe endpointslice -n prod
kubectl get pods -n prod -l app=payment-service -o wide

13.3 Node Forwarding Layer

kubectl -n kube-system get cm kube-proxy -o yaml
ipvsadm -Ln
iptables-save | grep payment-service

13.4 Client Connection Layer

Investigate connection pools, DNS cache, stale connections, retry storms. Observe thread pools, HTTP client pools, gRPC channel states, TIME_WAIT / CLOSE_WAIT counts.

13.5 Governance Layer

Detect amplified issues from timeouts, retries, or aggressive readiness failures. Ensure policies are not unintentionally magnifying faults.

Production Optimization Checklist

Platform Layer

Prefer IPVS for kube‑proxy; evaluate Cilium eBPF.

Enable EndpointSlice.

Run 2‑3 CoreDNS replicas with monitoring.

Enable NodeLocal DNSCache for large clusters.

Configure PDB for CoreDNS, Ingress, Gateway.

Use PodAntiAffinity and topologySpreadConstraints.

Deploy critical entry points across multiple AZs.

Application Layer

Explicitly set connection timeout, request timeout, total timeout.

Configure connection pool parameters; avoid defaults.

Control DNS TTL and connection lifetimes.

Graceful shutdown must first fail readiness.

Retries must be idempotent, back‑off, and limited.

Refresh long‑lived connections periodically.

Release Layer

RollingUpdate with maxUnavailable: 0.

PDB for core services.

Canary releases via Gateway or Mesh.

Validate DNS, Service, and Gateway configs in pre‑release environment.

Observability Layer

Key metrics (at least): CoreDNS QPS/latency/cache‑hit, EndpointSlice change rate, kube‑proxy sync delay, Service 5xx/timeout/connection count, Pod readiness volatility, client connection‑pool usage.

sum(rate(coredns_dns_requests_total[1m]))

histogram_quantile(0.99, sum(rate(coredns_dns_request_duration_seconds_bucket[5m])) by (le))

sum(rate(nginx_ingress_controller_requests{status=~"5.."}[1m])) by (ingress)

Anti‑Patterns

Using liveness as readiness – causes instant restarts on temporary unavailability.

Blind unlimited retries – exponential traffic amplification.

Relying on default DNS cache – stale records keep hitting failed instances.

Exposing every service via LoadBalancer – high cost, security surface, fragmented routing.

Treating Headless Service as universal – client complexity, inconsistent SDK behaviour.

Migration Path

Solidify native Kubernetes discovery (Service, CoreDNS, EndpointSlice, probes, rolling updates).

Platform‑level ingress/gateway standardisation.

Introduce stronger traffic governance (gateway features, policy).

When scale or cross‑cluster needs demand, adopt Service Mesh or eBPF dataplane.

Conclusion – Service Discovery as a Chain of Capabilities

A mature Kubernetes service discovery provides rapid inclusion of new instances, swift removal of faulty ones, prevents long‑connection amplification, aligns entry‑point and internal call strategies, synchronises scaling and release with discovery, and remains stable across multi‑cluster and multi‑AZ deployments.