Building a Stable High‑Traffic Microservice System with Consul‑Access and Multi‑Tenant Service Discovery

Overview

As businesses scale, many small‑to‑medium companies face rapid growth that outpaces their existing architecture, leading to unstable microservice systems. This guide, based on a talk by Tencent expert Liu Zhixin, shows how to use open‑source components to build a stable, high‑traffic microservice platform, focusing on service discovery, multi‑tenant support, and performance improvements.

Microservice Platform Architecture

The platform consists of multiple layers: a multi‑tenant VPC layer, a service registration & discovery module, an application management module (handling CI/CD, deployment, and configuration), and a data‑driven operations module for metrics, logs, and tracing. The data plane (blue box) handles the core request flow, while the control plane (gray area) simplifies microservice usage.

Service Discovery Basics

In a cloud era, microservices are small, independently deployable units that communicate via REST APIs. Service discovery is essential because container IPs change on each start. Consul, Zookeeper, and Nacos provide registration and discovery, with Consul being a popular choice for its HTTP API and simplicity.

Extending Consul with an Access Layer

Consul’s native API lacks multi‑tenant and namespace support. An access layer is introduced between clients and the Consul server to translate registration, heartbeat, and discovery requests into KV operations that embed tenant information. This layer stores instance data under keys like /tenant/service/serviceA/instance-id/data, enabling multi‑tenant isolation without modifying the original Consul API.

Three‑Step Registration Flow

Service instance registers itself.

Instance sends periodic heartbeats.

Clients pull the list of providers.

The access layer intercepts each step, converting API calls to KV writes/reads, merging heartbeat timestamps, and marking nodes as critical if no heartbeat is received within 30 seconds.

Token‑Based Tenant Identification

Because native Consul registration does not carry tenant data, a token module is added. Clients obtain a token containing tenant information from a token‑server; the access layer validates the token and extracts the tenant ID, making tenant handling transparent to the client.

Performance Optimizations

Adding the access layer reduces the number of long‑lived connections to Consul servers. For example, with 100 service instances each subscribing to two services, the direct approach creates 200 client‑to‑Consul connections, while the access layer aggregates them to only a few connections per access node. This dramatically lowers the load on Consul leaders, which otherwise must iterate over all watch connections for each change.

Additional optimizations include:

Aggregating watch requests to reduce CPU usage.

Extending watch timeouts (e.g., converting 2 s watches to 55 s on the access side) to cut CPU consumption by ~60 %.

Horizontal scaling of the stateless access layer to spread load.

High‑Availability Enhancements

Consul is a CP system and cannot guarantee continuous availability under network partitions. To improve resilience:

Clients use watch‑based discovery instead of periodic polling.

Local caches store the last known service list, providing fallback when Consul is unavailable.

Zero‑instance protection prevents replacing a healthy cache with an empty list.

Heartbeat read‑timeout is reduced (e.g., to 5 s) and retries are added to avoid long outages caused by missed heartbeats.

An access‑layer health‑check agent monitors Consul and access nodes, raising alerts on failures.

Graceful Shutdown and Deployment

To avoid request failures during service removal, a pre‑stop hook in Kubernetes performs deregistration, waits ~35 s, then shuts down the container. The access layer suppresses heartbeats from deregistered instances during this window.

For rolling updates, readiness probes query Consul to confirm the instance is registered before marking the pod as ready, ensuring traffic is only sent to fully initialized services.

Advanced Service Call Stability

Beyond discovery, the article discusses adding resilience to inter‑service calls:

Insert Hystrix (or Resilience4j) circuit breakers and fallback mechanisms between Feign and Ribbon.

Configure sensible connection and read timeouts (e.g., 5 s connect, 3–5 s read for fast APIs, up to 1 min for heavy queries) and limited retries.

Implement instance‑level circuit breaking using Resilience4j to quickly isolate problematic nodes.

Conclusion

By layering an access component and token service on top of Consul, introducing performance‑focused aggregations, and applying robust high‑availability and deployment practices, a microservice ecosystem can achieve both functional stability and operational scalability.