Designing Rate Limiting and Circuit Breaking for Microservices and API Gateways

Microservice architectures and API gateways often face problems such as a single API causing resource exhaustion or cascading failures (snowball effect). To prevent these issues, rate limiting and circuit breaking mechanisms are introduced to isolate problematic services and protect overall system stability.

Basic Concepts

Rate limiting queues requests and caps the number of concurrent threads, while circuit breaking makes an entire service unavailable when certain thresholds are breached. The article distinguishes between limiting at the consumer‑API level and circuit breaking at the provider‑service level.

Resource Granularity

Three granularity levels are proposed: the finest grain (consumer + API + provider), the circuit‑break layer (API + provider), and the circuit‑break scope (provider only). Choosing the appropriate granularity influences rule configuration and real‑time data aggregation.

Rule Definition

Rules consist of a resource, a metric (e.g., request count, latency, failure rate), and a threshold. Simple rules trigger when a single metric exceeds a limit; composite rules combine multiple conditions with logical AND/OR. Rules can target specific consumer‑API pairs or all consumers of a service.

Calculation Logic

Metrics are collected in short intervals (e.g., 10 seconds) and first‑aggregated, then stored in a sliding window. A second aggregation over the window (e.g., 5 minutes) evaluates the defined rules to decide whether to apply rate limiting or circuit breaking.

Implementation Flow

Match incoming requests to resources based on rule configuration and store raw data in a temporary buffer.

Perform first‑level aggregation and push results into the sliding window.

Periodically aggregate window data, evaluate rules, and trigger limiting or breaking actions.

When a service is broken, optionally set a recovery timer and schedule a task to restore normal operation once conditions improve.

Decoupling from API Gateway

The rate‑limiting/circuit‑breaking component acts as an independent interceptor that can be plugged into any API gateway or microservice framework, allowing the gateway to remain lightweight while the interceptor enforces protection policies.

Overall, the article provides a practical, fine‑grained approach to designing rate limiting and circuit breaking solutions that can be adapted to various microservice environments.