Designing a Scalable API Gateway for Microservices: Architecture and Key Features

Overall Architecture

With the rise of microservices, an API gateway has become an indispensable component, serving as the sole entry point for external traffic while extracting common functionalities for applications.

The current gateway sits at the top of the traffic flow, handling all north‑south traffic and distributing requests to microservice or BFF aggregation applications. Before BFF standardization, it is treated as a regular microservice.

Request Dispatch

Request dispatch is the most basic gateway function. Unlike Nginx‑based gateways that update upstream dynamically, our gateway treats itself as a subscriber‑only microservice. It receives only URL information, reusing existing service‑discovery capabilities after URL normalization and prefix handling.

Service discovery objects are managed via a control platform, allowing dynamic addition or removal of services through hot‑updates of in‑memory caches.

Conditional Routing

Conditional routing enables filtering requests based on content or traffic proportion, supporting gray releases, blue‑green deployments, and A/B testing. Our implementation reuses the existing microservice framework: each application defines groups of instances, and the gateway creates Invoker objects that contain Space objects for each group. Requests are matched against these spaces to determine routing.

We currently support matching by specific content (e.g., HTTP headers, attributes) and by traffic proportion, primarily using a "version" label for instance grouping.

API Management

The gateway includes API management to distinguish internal microservice calls from external client calls. API definitions are stored in a database; publishing actions on the control platform trigger Pub/Sub notifications to the gateway. A filter layer removes unpublished or offline APIs during namespace matching, enabling hot updates.

Synchronize newly added APIs from the microservice control platform to the gateway without manual steps; support multiple import formats.

Publish APIs across environments via a flow feature.

Filter and display APIs by status.

Integrate with DevOps platforms to remind owners of API publishing tasks.

Rate Limiting & Circuit Breaking

Given the high concurrency of north‑south traffic, the gateway implements rate limiting, isolation, and circuit breaking using a stability platform built on Sentinel. Limits control QPS and concurrent requests; circuit breaking triggers on error‑rate thresholds and returns fallback data.

All rules are configured in the stability platform, propagated via the configuration center, and applied in real time. Metrics are exposed to Prometheus for monitoring.

Security Policies

To mitigate malicious traffic such as crawlers, the gateway provides configurable security rules. Users can manually set rules via the control platform, which are pushed to the gateway for hot updates. Additionally, logs are streamed to a big‑data analysis platform that can automatically generate block rules and trigger alerts.

Supported rule dimensions include client IP, user ID, and arbitrary HTTP headers/attributes, with actions like fast‑fail or CAPTCHA challenges.

Monitoring, Alerting & Tracing

The gateway integrates monitoring, alerting, and tracing without code changes. Metrics SDK exposes data to a monitoring endpoint, while tracing SDK records call chains. Both metrics and tracing logs are collected by a log collector and visualized in the monitoring platform. Dual‑side alerts notify both gateway owners and downstream service owners when failures occur.

Summary & Future Work

The gateway also supports WebSocket protocols and acts as a multi‑cloud traffic scheduler. Future improvements include validating high‑concurrency performance, introducing a rule engine for security, stability, and routing policies, and expanding security rule capabilities (e.g., IP ranges, logical operators).