Designing a High‑Performance Microservice Gateway: Routing, Load‑Balancing & Resilience

Functional Design Overview

The gateway serves as the entry point for microservices, handling routing, load‑balancing, aggregation, authentication, overload protection, circuit breaking, service degradation, caching, retries, logging, and management.

Routing

Typical RESTful services are routed based on host, URL, and other rules. To enable hot‑deployment of routing rules, two main approaches are considered:

Database‑based routing : Store routing rules in a database, query them at request time, and optionally cache them (e.g., in Redis). This requires synchronization logic between the database and cache and an admin UI.

Configuration‑file routing : Load static configuration files at startup. Dynamic updates can be achieved via a configuration server, reducing development effort compared to the database approach.

Load Balancing

Common algorithms include random, weighted random, round‑robin, weighted round‑robin, least‑connections, and source‑address hash. For microservice scenarios, source‑address hash is preferred because it avoids session‑sharing issues, is simple to implement, and offers good cost‑performance when traffic is moderate.

Aggregation Services

Two aggregation strategies are discussed:

GraphQL : Provides a query language for APIs. Options include adding a dedicated GraphQL aggregation server before the gateway, embedding GraphQL directly in the gateway (requires restart), or adding an aggregation layer after the gateway.

Coding : Implement aggregation logic in the gateway itself, assembling responses from multiple services. This approach is favored due to lower learning cost and limited aggregation needs.

Authentication & Authorization

The system adopts Role‑Based Access Control (RBAC). RBAC0, the simplest model, is sufficient for typical internet projects, and Spring Security is chosen as the authentication framework for its integration convenience.

Overload Protection

Rate limiting is the most suitable traffic‑control method for microservices. Token‑bucket algorithms are recommended, with leaky‑bucket as an alternative for strict bandwidth control. Implementation can be placed at the ingress layer (e.g., Nginx) when custom logic is unnecessary.

Circuit Breaker

When failure counts exceed a threshold, the circuit opens, returning errors immediately. After a cooldown (typically the mean time to recovery), the circuit enters a half‑open state, allowing limited requests to test recovery before fully closing.

Service Degradation

A queue‑based approach is suggested: incoming requests enter a fixed‑size blocking queue; if the queue exceeds a threshold, degradation rules determine whether to reject the request or continue queuing.

Caching

Use a centralized cache (e.g., Redis) for all gateway‑level data. The cache workflow includes lookup, cache hit return, cache miss forward to target service, and caching of the response. Beware of cache avalanche, penetration, and breakdown.

Service Retry

Retry logic requires configuration (which endpoints, retry count, timeouts) and execution (performing the retries). Dynamic configuration and proper timeout calculations are essential.

Logging

Follow project logging standards. Access logs can initially be captured at the ingress layer (e.g., Nginx) and refined later based on specific needs.

Non‑Functional Design

High Performance

Thread‑per‑request models suffer from context‑switch overhead. The gateway should adopt a Reactor model, typically implemented with Netty, to achieve scalable I/O handling.

Refer to "EDA Style and Reactor Pattern" for details on the Reactor model.

High Availability

Include graceful shutdown, slow‑start for newly started instances, and support for service graceful deregistration. Both the downstream services and the gateway itself should finish in‑flight requests before terminating.

Scalability

Design the gateway as a stateless service, using token‑based authentication to decouple user state. Tokens are issued at login, stored with expiration, and validated on each request.

Extensibility

Implement global and business‑specific interceptors, ordered by priority. Provide dynamic configuration via API calls or a configuration server to enable/disable interceptors per request pattern.

Observability

Leverage existing monitoring solutions such as SkyWalking or Pinpoint for tracing, and ELK stack for log aggregation and analysis.