How Kubernetes Service Discovery Keeps Your Pods Connected

K8S Service Discovery Journey

Kubernetes service discovery is a topic that often causes confusion. This article is divided into two parts: background networking knowledge and an in‑depth look at Kubernetes service discovery.

K8S Network Basics

Before exploring service discovery, understand these points:

Kubernetes applications run in containers inside Pods.

Each Pod attaches to a flat IP network, often a VXLAN overlay, called the Pod network.

Each Pod has a unique, routable IP address within the Pod network.

These three factors allow each application component to communicate directly without NAT.

Dynamic Network

When scaling horizontally, new Pods with new IPs are added; when scaling down, Pods and their IPs are removed, creating apparent chaos. Rolling updates add new Pods and remove old ones, changing the set of IPs. Without a stable mechanism, each service would need to monitor the network and maintain a healthy Pod list, which is inefficient. Kubernetes solves this with the Service object.

Calling this object "Service" can be confusing because the term also describes application processes.

Service Provides Stability

A Kubernetes Service creates a stable network endpoint in front of a group of Pods and load‑balances traffic to them.

Typically a Service sits in front of a set of Pods that perform the same work, e.g., a Service for web front‑ends and another for authentication services.

Clients communicate with the Service, which distributes traffic to the Pods.

The Service’s name, IP, and port remain constant even as underlying Pods change.

K8S Service Breakdown

A Service consists of a stable front end (name, IP, port) and a dynamic back end (the set of Pods matching a label selector).

The front end never changes, preventing client DNS cache issues. The back end is a dynamic group of Pods accessed via load balancing.

The load balancer operates at layer 4 with simple round‑robin, keeping connections consistent.

Service Registration

Kubernetes uses DNS as the service registry. Each cluster runs a DNS pod (CoreDNS) in the kube‑system namespace. Every Service automatically registers a DNS A record.

POST a new Service definition to the API server.

The request passes authentication, authorization, and admission checks.

The Service receives a ClusterIP (virtual IP) and is stored in the cluster data store.

The Service configuration propagates cluster‑wide.

CoreDNS creates the corresponding DNS A record.

CoreDNS watches the API server for new Service objects and creates the DNS entries, so Services do not need to register themselves.

Endpoint Object

After a Service’s front end registers, the back end maintains a list of Pods. The Service’s label selector determines which Pods belong to the Service, and Kubernetes automatically creates an Endpoints object containing the IPs of those Pods.

Service Discovery

Assume two applications, my‑app and your‑app, each with a Service ( my‑app‑svc and your‑app‑svc) that have ClusterIP addresses 10.0.0.10 and 10.0.0.20 respectively.

Pods resolve Service names via the cluster DNS, obtaining the ClusterIP, then send traffic to that IP.

The client must know the target Service’s name.

Network

When a Pod sends traffic to a Service’s ClusterIP, the packet enters the Service Network, which has no direct routes. The packet is first sent to the node’s default gateway (cbr0 bridge), then to the node’s network interface.

Each node runs kube‑proxy, which watches Service objects and creates iptables or IPVS rules. These rules capture traffic destined for a Service’s ClusterIP and rewrite the destination to the IP of a healthy Pod from the Endpoints object.

iptables is being replaced by IPVS (Kubernetes 1.11+), which offers better performance for layer‑4 load balancing.

Summary

Creating a Service allocates a virtual ClusterIP, registers the name and IP in cluster DNS, and creates an Endpoints object that holds the list of healthy Pods matching the Service’s selector. All nodes configure iptables/IPVS rules so that traffic to the ClusterIP is rewritten to the appropriate Pod IP, enabling reliable intra‑cluster communication.