Designing a High‑Performance API Gateway for Microservices: Architecture & Key Features

Overall Architecture

With the rise of microservices, API gateways have become an indispensable component, serving as the sole entry point for external traffic and extracting common functionalities for applications.

The gateway sits at the top, handling all north‑south traffic, and forwards requests downstream to microservice or BFF aggregation applications. Before BFF standardization, it is treated as a regular microservice.

The gateway implements request dispatch, conditional routing, API management, rate limiting & isolation, circuit breaking, security policies, monitoring & alerts, and call‑chain tracing.

It is built on RxNetty, providing an asynchronous, reactive processing pipeline that achieves high concurrency on a single machine. Most features are assembled from existing platforms: request dispatch and conditional routing use the microservice framework; rate limiting, isolation, and circuit breaking rely on a stability platform; monitoring uses a monitoring platform; security policies leverage a big‑data analysis platform. Service registration and configuration are handled by separate registry and config centers.

Request Dispatch

Request dispatch is the gateway’s most basic function. Unlike Nginx‑based gateways that dynamically update upstreams, our gateway treats itself as a non‑registering microservice that receives only URL information. After URL normalization, each application registers a unique prefix in the registry, allowing the gateway to construct namespace objects for each prefix and match incoming requests to the appropriate microservice.

Dynamic updates are achieved by managing service discovery objects via a control platform; changes are pushed through the configuration center, triggering hot updates of the in‑memory cache.

Conditional Routing

Conditional routing enables routing of requests with specific content or traffic ratios to particular instance groups, supporting gray releases, blue‑green deployments, and A/B testing.

Our implementation reuses the existing microservice framework: each application defines groups with instances, and the gateway creates Invoker objects that contain Space objects for each group. During request handling, the gateway matches rules against these spaces to decide routing, without additional development.

We currently support matching based on request content (e.g., headers, attributes) or traffic proportion, primarily using a “version” label for grouping, with future plans to support Kubernetes pod labels.

API Management

The gateway also provides API management, distinguishing internal microservice calls from external client calls. API definitions are stored in a database; publishing actions on the control platform are propagated via the configuration center. A filter layer removes unpublished or deleted APIs during namespace matching, enabling hot updates.

Key user‑experience improvements include:

Automatic synchronization of newly added APIs from the microservice control platform to the gateway without manual steps, supporting multiple import formats.

Environment‑wide API publishing to avoid duplicate entries.

Filtering and displaying APIs by status.

Integration with DevOps pipelines to trigger API management updates during application releases.

Rate Limiting & Isolation / Circuit Breaking

Given the gateway’s role as the sole entry for north‑south traffic, high concurrency and complex traffic patterns necessitate robust flow control.

We rely on a stability platform (a customized Sentinel) and the configuration center. Rate limiting controls QPS, isolation limits concurrent requests, and circuit breaking triggers fallback responses when error rates exceed thresholds. All rules are configured on the stability platform, distributed via the configuration center, and applied through hot updates. Metrics are exposed via Prometheus for monitoring.

Security Policies

To mitigate malicious traffic such as web crawlers, the gateway implements configurable security policies. Rules can be manually set via the control platform and pushed through the configuration center to the gateway’s security control component, which evaluates incoming requests in real time.

Automated rule generation is also supported: gateway logs are streamed to a big‑data analysis platform, which detects anomalous IPs or users and automatically creates security rules, triggering alerts for owners.

Supported criteria include client IP, user ID, and various HTTP headers/attributes, with actions like fast‑fail or CAPTCHA challenges.

Monitoring, Alerts & Tracing

The gateway provides comprehensive monitoring, alerting, call‑chain tracing, and log querying. Metrics are exposed via an endpoint for the monitoring system, while tracing SDKs emit tracing logs. Both metrics and tracing data are collected by a log collector and processed by the monitoring platform.

Alerts can be configured for metric thresholds, error logs, or host‑level issues. When backend microservices experience failures (e.g., timeouts, connection pool exhaustion), the gateway generates alerts for both the gateway owner and the affected microservice owner via a dual‑alert mechanism.

Summary

The API gateway offers many additional capabilities not covered here, such as WebSocket support and multi‑cloud traffic scheduling. Future improvements include validating high‑concurrency performance, integrating a rule engine for security, stability, and routing policies, and extending security policies to support IP ranges and complex logical expressions.