Mastering Rate Limiting: Choosing the Right Algorithm for High‑Availability Systems

Rate Limiting Concept

Rate limiting controls the speed of concurrent requests or the number of requests within a time window to protect a system; when the limit is reached, requests can be rejected, queued, or degraded. It ensures high availability by keeping traffic below a defined threshold, preventing issues such as thread‑pool blockage or database overload.

Typical dimensions include IP‑based limits, QPS or concurrency limits per interface, and black‑/white‑list rules.

Rate Limiting Algorithms Overview

Fixed Window

A counter is maintained for each fixed time interval. If the count is below the threshold, the request is allowed and the counter increments; otherwise it is rejected. This simple approach can cause bursty traffic to be either fully accepted or fully rejected within a window.

Sliding Window

An improvement over the fixed window, the sliding window counts requests in the most recent interval ending at the current time, smoothing out bursty traffic and preventing spikes from exceeding the limit.

Leaky Bucket

Requests flow into a bucket at any rate but exit at a constant rate, smoothing bursts into a steady stream. Excess requests overflow and are dropped, providing a strict rate guarantee.

Token Bucket

Tokens are added to a bucket at a fixed rate; each request consumes a token. If tokens are unavailable, the request is rejected. This allows short bursts while maintaining an average rate, and is used by implementations such as Guava RateLimiter.

Trade‑offs and Summary

Fixed Window: Simple but cannot handle burst traffic; useful as a fallback.

Sliding Window: Better burst handling; moderate implementation complexity.

Leaky Bucket: Guarantees steady output; less flexible for sudden spikes.

Token Bucket: Handles bursts efficiently but requires careful token‑bucket sizing.

Implementation difficulty varies: fixed and sliding windows are easy to deploy in both single‑node and distributed environments, while leaky and token buckets are more common in single‑node scenarios and require additional coordination when used globally.

Voice‑Bot System Rate Limiting Design

The voice‑bot service is a B2B product serving enterprise tenants with varying usage patterns. To protect the system, a three‑layer throttling strategy was designed:

Tenant‑level global rate limiting (Redis‑backed).

Interface‑level single‑machine rate limiting (Sentinel).

Layer 1: Tenant‑Level Second‑Level Sliding Window

Using Redis sorted sets (zset) keyed by tenant ID and method, timestamps of requests are stored. A Lua script removes entries older than the current sliding window and counts the remaining items; if the count is below the threshold, the request passes, otherwise it is rejected.

Layer 2: Tenant‑Level Minute‑Level Sliding Window

To limit sustained traffic, a minute‑granularity sliding window is used. Each minute has its own counter bucket; the sum of the last N minute buckets is compared against a threshold derived from the second‑level limit.

Layer 3: Sentinel Single‑Machine Limiting

Alibaba Sentinel is integrated to provide fine‑grained per‑endpoint QPS or concurrency limits, acting as the final safeguard for each service instance.

Additional Optimization: IP‑Based Fixed Window

To reduce Redis pressure under heavy multithreaded load, an IP‑plus‑method fixed‑window limit (twice the tenant threshold) is applied locally, rejecting the majority of excess requests before they reach Redis.

Key Considerations

Threshold Setting: Determine safe limits via load testing; start with conservative values and adjust based on observed traffic.

Performance Impact: Rate limiting should add minimal overhead; excessive Redis calls can degrade overall system performance, so batch operations and Lua scripts are used.

Conclusion

Rate limiting is essential for high‑availability systems. No single algorithm is universally best; the choice depends on workload characteristics. For the voice‑bot platform, a sliding‑window strategy was selected for tenant‑level limits, complemented by token‑bucket‑style bursts via Sentinel and an IP‑based fixed window to protect Redis resources.