Demystifying Kubernetes Service Discovery: From Pods to Stable Endpoints

K8S Service Discovery Journey

Kubernetes service discovery is often confusing; this article is split into two parts: network background knowledge and an in‑depth look at Kubernetes service discovery.

K8S Network Basics

Before exploring service discovery, understand that:

Kubernetes workloads run in containers inside Pods.

All Pods share a flat IP network (usually a VXLAN overlay), called the Pod network.

Each Pod has a unique, routable IP address.

These three facts let application components communicate directly without NAT.

Dynamic Network

When scaling out, new Pods with new IPs are added; when scaling in, Pods and their IPs disappear, making the network appear chaotic. Rolling updates add new Pods and remove old ones, causing similar churn.

Calling this object "Service" is a bad idea because the word already describes a process or component.

Kubernetes also tracks used and available IPs (IPAM).

Service Brings Stability

A Service creates a stable network endpoint in front of a group of Pods and load‑balances traffic to them.

Typically you place a Service before a set of Pods that perform the same function, e.g., a front‑end Service for web Pods and another Service for authentication Pods.

Clients talk to the Service; the Service distributes traffic to the Pods.

The Service’s name, IP, and port never change even as underlying Pods are added or removed.

K8S Service Parsing

A Service consists of:

Front‑end: immutable name, IP, and port.

Back‑end: a set of Pods selected by label selectors.

The front‑end’s stability avoids DNS cache issues for clients.

The back‑end is dynamic; the selected Pods are load‑balanced.

Load balancing is a simple 4‑layer round‑robin; all connections from the same client go to the same Pod.

Service Registration

When a Service is created, it is automatically registered in the cluster DNS (CoreDNS) as an A record mapping the Service name to its ClusterIP.

POST a new Service definition to the API Server.

The request passes authentication, authorization, and admission checks.

The Service receives a ClusterIP and is stored in the cluster data store.

The Service configuration is propagated cluster‑wide.

CoreDNS creates the corresponding DNS A record.

Other Pods can now discover the Service via DNS.

Endpoint Object

Each Service has a Label Selector; Kubernetes automatically creates an Endpoints object that holds the list of Pods matching the selector.

The Endpoints object is kept up‑to‑date so the Service always forwards traffic to healthy Pods.

Service Discovery in Action

Assume two applications, my‑app and your‑app, each with its own Service ( my‑app‑svc and your‑app‑svc) having ClusterIPs 10.0.0.10 and 10.0.0.20 respectively.

Pods query the cluster DNS to resolve a Service name to its ClusterIP, then send traffic to that virtual IP.

Clients must know the target Service name.

Network Path

ClusterIP belongs to the Service Network, which has no direct routes. Traffic is first sent to the node’s default bridge (cbr0), then to the node’s network interface, where kube‑proxy rewrites the packet’s destination to a healthy Pod IP.

kube‑proxy does not proxy traffic; it only programs iptables or IPVS rules.

kube‑proxy watches Service objects, receives the ClusterIP and associated Endpoints, and creates the necessary iptables/IPVS rules on each node.

IPVS is replacing iptables as the preferred 4‑layer load balancer in newer Kubernetes versions.

Summary

Creating a Service yields a virtual ClusterIP that is registered in the cluster DNS and linked to an Endpoints object containing the current set of healthy Pods. All nodes install iptables/IPVS rules so that traffic destined for the ClusterIP is rewritten to the appropriate Pod IP, enabling stable, load‑balanced communication between Pods.