Rate Limiting & Circuit Breaking: Algorithms and Practical Strategies

1 Why Rate Limiting

In distributed systems, services often depend on multiple downstream services. A synchronous call to an unavailable inventory service can block the product service thread, leading to resource exhaustion and a cascade failure known as the avalanche effect.

1.2 Common Avalanche Scenarios

Hardware failures such as server crashes, power outages, or fiber cuts.

Traffic spikes caused by abnormal traffic, promotional events, or retry amplification.

Cache penetration when caches restart and miss, flooding the database.

Program bugs like infinite loops, memory leaks, deadlocks, or prolonged Full GC.

Synchronous waiting that exhausts resources.

1.3 Strategies to Prevent Avalanche

Hardware failures: multi‑zone or multi‑data‑center redundancy.

Traffic spikes: dynamic scaling and traffic control (circuit breaking, rate limiting, timeout retries).

Cache penetration: cache pre‑loading, asynchronous loading, random expiration.

Program robustness: fix bugs, avoid leaks, release resources promptly.

Synchronous waiting: resource isolation, MQ decoupling, fast‑fail on unavailable services.

Overall, services must isolate their dependencies and failures to avoid becoming a point of avalanche; self‑protection mechanisms like circuit breaking, rate limiting, and exception eviction are essential.

2 Common Rate‑Limiting Algorithms

2.1 Counter‑Based Limiting

The counter algorithm records request counts within a fixed time window; when the count exceeds the limit, further requests are rejected. Implementations can be in‑memory (e.g., Atomic) for single‑node or use Redis INCR/DECR for distributed scenarios.

Potential issue: uneven request distribution can cause the limit to be exhausted early in the window, leaving idle capacity later.

2.2 Fixed Window Limiting

This method divides time into fixed windows (e.g., 1 s). Each request increments a counter; once the limit is reached, subsequent requests in that window are denied. The counter resets at the end of each window.

Divide time into fixed windows.

Increment counter for each request within the window.

Reject requests after the limit is reached.

Reset counter when the window ends.

2.3 Sliding Window Limiting

Sliding windows record timestamps of each request, ensuring that any sliding interval does not exceed the threshold, at the cost of higher memory usage.

Partition a window into fine‑grained slots, each with its own counter.

When a slot expires, drop its count and add a new slot.

Sum counters across slots; if total exceeds limit, reject excess requests.

2.4 Leaky Bucket Algorithm

The leaky bucket releases requests at a constant rate, smoothing bursts. Incoming requests are queued in the bucket; if the bucket is full, excess requests are dropped.

Enqueue arriving requests.

Reject when bucket is full.

Process requests at a fixed outflow rate.

2.5 Token Bucket Algorithm

Tokens are added to the bucket at a steady rate; a request proceeds only if it can acquire a token. When tokens run out, requests are rejected. This allows short bursts while enforcing an average rate.

Tokens are added at a fixed rate.

Excess tokens are discarded when the bucket is full.

Each request consumes one token; if none are available, the request is rejected.

2.6 Practical Usage

Google Guava’s RateLimiter implements a token‑bucket with warm‑up support. Alibaba’s Sentinel uses a leaky‑bucket‑based uniform‑queue strategy.

3 Main Circuit‑Breaker and Rate‑Limiting Solutions

3.1 Sentinel Circuit Breaking

3.1.1 Overview

Sentinel is a high‑availability traffic‑management framework for distributed systems. It can limit QPS (e.g., 10,000 requests per second) and provide graceful degradation when thresholds are exceeded.

3.1.2 Features

Rich scenarios: flash sales, message throttling, cluster traffic control, real‑time downstream circuit breaking.

Real‑time monitoring of per‑instance and cluster metrics.

Extensive ecosystem integration (Spring Cloud, Dubbo, gRPC, etc.).

SPI extension points for custom rule management and dynamic data sources.

3.2 Hystrix (Spring Cloud)

Hystrix is an open‑source library for latency and fault tolerance in distributed systems. It provides circuit‑breaker functionality that returns fallback responses instead of blocking threads, preventing cascading failures.

3.2.1 Usage

Implement HystrixCommand and invoke via execute(), queue(), observe(), or toObservable().

Annotate fallback methods with @WafHystrixFallback.

Check circuit state; handle thread‑pool, queue, or semaphore saturation; trigger fallback on failure or timeout.

4 Conclusion

Rate limiting and circuit breaking protect services from overload, latency spikes, and avalanche failures.

Common algorithms: counter, fixed window, sliding window, leaky bucket, token bucket.

Popular implementations: Sentinel, Hystrix, Guava RateLimiter.