Designing High‑Availability for Microservices: Service Discovery & Config Management Best Practices

Introduction

A customer deployed many microservices on an Alibaba Cloud Kubernetes cluster; a single node’s network card failed, causing DNS resolution problems and making downstream services unavailable.

Risk Chain Analysis

ECS fault node hosted all CoreDNS pods without spreading, breaking cluster DNS.

The client used a buggy nacos-client 1.4.1 version whose heartbeat stopped after DNS failure, requiring a restart to recover.

Alibaba announced the bug in May, but the customer missed the notice and kept using the vulnerable version in production.

The cascade of these risks led to service unavailability and business loss.

High‑Availability Design Principles

Limit impact scope : Deploy at least three replicas, use multiple availability zones, and automatically isolate faulty nodes.

Shorten failure duration : Implement observability (e.g., Prometheus alerts) and establish rapid emergency response procedures.

Reduce exposure frequency : Adopt gray releases, limit releases during peak events, and avoid unnecessary deployments.

Lower failure probability : Upgrade architectures (e.g., Nacos 2.0) to improve data partitioning and long‑connection models.

Service Discovery High‑Availability

Service discovery consists of Consumer and Provider sides. Risks include Provider heartbeat DNS failures, registration loss, and empty instance lists. Mitigations:

Push‑empty protection : When a Consumer receives an empty list, it falls back to a local cache instead of failing.

Service degradation, rate limiting, circuit breaking : Reduce heartbeat intervals, limit non‑core functions, and protect traffic capacity.

Enable push‑empty protection in spring.cloud.nacos.discovery.namingPushEmptyProtection=true (available from nacos-client 1.4.2 onward) and configure cache directories at ${user.home}/nacos/naming/${namespaceId}.

Configuration Management High‑Availability

Configuration management handles subscription and publishing across multiple environments. High‑availability tactics include:

Multi‑replica configuration center instances across zones.

Graceful gray releases, versioned rollbacks, and audit trails.

Separate disaster‑recovery and cache directories on the client; the disaster directory is consulted first when the server is unavailable.

Rate‑limit read/write operations and enforce connection limits to protect the configuration center under load.

Hands‑On Practice

Environment Preparation

Purchase an MSE registration & configuration center (Professional edition) and an MSE cloud‑native gateway. Pre‑configure downstream relationships (A→C, B→C).

Application Deployment

Deploy three services (A, B, C) on ACK using the following Kubernetes manifests (excerpt):

# A application (push‑empty protection enabled)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-a-b
  labels:
    app: spring-cloud-a
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-a
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-a
      labels:
        app: spring-cloud-a
    spec:
      containers:
      - name: spring-cloud-a
        image: mse-demo/demo:1.4.2
        env:
        - name: spring.cloud.nacos.discovery.server-addr
          value: mse-xxx-nacos-ans.mse.aliyuncs.com:8848
        - name: spring.cloud.nacos.discovery.namingPushEmptyProtection
          value: "true"
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: 250m
            memory: 512Mi
---
# B and C deployments omitted for brevity

Register Services in the Gateway

After deployment, register service A in the MSE cloud‑native gateway so that the gateway only calls A.

Verification and Adjustment

Use curl http://${gatewayIP}/ip to verify the call chain. Initially A calls C; then update configuration to make A call B, resulting in the chain A→B→C.

Continuous Traffic Simulation

while true; do sleep .1; curl -so /dev/null http://${gatewayIP}/ip; done

Fault Injection

Apply a Kubernetes NetworkPolicy to block B’s outbound traffic to the registration center (port 8848), causing its heartbeat to fail and the instance to be removed after ~30 seconds.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: block-registry-from-b
spec:
  podSelector:
    matchLabels:
      app: spring-cloud-b
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
    ports:
    - protocol: TCP
      port: 8080

Observation

Monitor the gateway’s success rate dashboard. After B is removed, the success rate drops to ~50 % and stabilizes, confirming that push‑empty protection on A keeps the call path alive.

Summary

The demo reproduces a realistic risk scenario and shows how client‑side high‑availability (push‑empty protection) together with robust service‑discovery and configuration‑management designs can contain failures, maintain service continuity, and protect business operations.