Implement Region‑Based Failover with Istio on Kubernetes: Step‑by‑Step Guide

Background

As applications grow and become more complex, the number of microservices increases, raising the probability of failures caused by hardware issues, network latency, software bugs, or human error. Failover is a fundamental resilience capability that ensures the system continues operating and recovers with minimal impact, improving overall availability.

Introduction

Cloud‑native Kubernetes (K8s) and Istio use specific node labels as locality information:

Region: a large geographic area (e.g., us‑east). The label topology.kubernetes.io/region determines the node’s region.

Zone: a group of compute resources within a region. The label topology.kubernetes.io/zone determines the node’s zone.

Subzone (partition): Istio introduces a custom node label topology.istio.io/subzone to define finer‑grained partitions such as a rack. It can be inspected with kubectl describe node xxx | grep topo .

The demonstration environment deploys multiple instances of a helloworld service across different zones (different topology.kubernetes.io/zone labels). By configuring localityLbSetting.failover and outlierDetection in an Istio DestinationRule, traffic is first directed to services in the same zone, and if all instances in that zone fail, traffic is fail‑overed to a specified zone.

Hands‑on Exercise

Deploy the server

Create a sample namespace with automatic Istio sidecar injection.

apiVersion: v1
kind: Namespace
metadata:
  name: sample
  labels:
    istio-injection: enabled

Deploy the helloworld service using a generated YAML script. The script produces a Service and a Deployment YAML that include a node affinity on topology.istio.io/subzone and a container exposing port 5000.

#!/bin/bash
set -euo pipefail
# (script content omitted for brevity)

Run the script for each location and apply the manifests:

for LOC in "beijing" "tianjin" "shenyang"; do ./genHelloWorld.sh --version "$LOC" > "helloworld-${LOC}.yaml"; done
kubectl apply -f helloworld-xxx.yaml -n sample

Deploy the client

Apply the sleep workload:

# Sleep service
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true

Inspect the client’s cluster information with:

kubectl exec -it sleep-xxx -c istio-proxy -n sample -- curl localhost:15000/clusters

Configure server‑side regional failover

Istio traffic management uses VirtualService, DestinationRule, and EnvoyFilter. Regional failover is defined in a DestinationRule with localityLbSetting.failover and outlierDetection:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: helloworld
  namespace: sample
spec:
  host: helloworld.sample.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        maxRequestsPerConnection: 1
    loadBalancer:
      simple: ROUND_ROBIN
      localityLbSetting:
        enabled: true
        failover:
        - from: cn-north-4
          to: cn-south-1
    outlierDetection:
      consecutive5xxErrors: 1
      interval: 1s
      baseEjectionTime: 1m

This policy performs outlier detection (isolating a pod after a single 5xx error) and fails over traffic to the cn‑south‑1 region when all instances in cn‑north‑4 become unhealthy.

Validate regional load balancing

From a Sleep pod located in cn‑north‑4/cn‑north‑4b/tianjin, multiple calls to helloworld consistently reach the same pod, confirming zone‑aware routing.

Validate regional failover

Simulate a failure by draining the Envoy listener on a helloworld pod:

kubectl exec helloworld-tianjin-xxx -n sample -c istio-proxy -- curl -sSL -X POST 127.0.0.1:15000/drain_listeners

Subsequent requests from the same Sleep pod are round‑robin‑distributed to helloworld instances in cn‑north‑4/cn‑north‑4b/beijing and cn‑north‑4/cn‑north‑4b/shenyang, demonstrating that traffic is rerouted to other zones when the original zone fails.

Notes

Failover configured here controls cross‑region scenarios; intra‑region zone or subzone failover occurs automatically. The example focuses on intra‑region operations and is typical for multi‑cluster environments.