Master Distributed Rate Limiting with Token Buckets, Redis, and Code

1. Microservice Rate Limiting

With the rise of microservices, the stability between services becomes increasingly critical. Caching, degradation, and rate limiting are three powerful tools to protect system stability.

Caching improves access speed and capacity, degradation temporarily disables problematic services, but some scenarios—such as scarce resources, database writes, or complex queries—cannot be solved by caching or degradation, so rate limiting is needed.

For example, a function may handle up to 3,000 QPS per second; if traffic exceeds this, a throttling mechanism must protect the system.

The goal of rate limiting is to restrict concurrent requests or the request rate within a time window, rejecting or delaying excess traffic once the limit is reached.

2. Rate‑Limiting Algorithms

2.1 Leaky Bucket Algorithm

The leaky bucket lets requests (water) flow into a bucket that leaks at a constant rate; excess requests overflow and are rejected, enforcing a strict transmission rate.

2.2 Token Bucket Algorithm

The token bucket works in the opposite direction, allowing bursts while controlling the average rate. Tokens are added to the bucket at a fixed interval (e.g., one token every 10 ms for 100 QPS). A request consumes a token; if none are available, the request is blocked or rejected.

2.3 Algorithm Choice

The leaky bucket strictly limits transmission speed, while the token bucket controls the average rate and permits bursts. The token bucket also allows easy adjustment of the refill speed, making it the preferred core algorithm for rate‑limiting frameworks.

3. Local Rate Limiting

Implementing a token bucket locally can use a Long as the bucket and AtomicLong for lock‑free CAS operations to add or remove tokens.

A scheduled thread continuously adds tokens to the bucket. The code for this scheduled token refill is shown below.

4. Distributed Rate Limiting Overview

Distributed rate limiting must address dynamic rules (changing QPS, enabling/disabling limits), cluster‑wide throttling across service instances, and circuit‑breaker degradation for unstable resources.

Additional optional features include real‑time monitoring, gateway flow control, hotspot parameter limiting, adaptive throttling, blacklist/whitelist, and annotation support.

5. Distributed Rate Limiting Solutions

1. Redis Token Bucket

Use Redis as a shared memory region for the token bucket when multiple instances exist; concurrency can be managed with Redis distributed locks. The downside is a network round‑trip for each token acquisition, limiting throughput.

2. Unified QPS Allocation

Distribute the total QPS among instances, possibly with weighted allocation based on server capacity. This approach still suffers from imbalance when traffic distribution does not match the static split.

3. Invoice Server (Control‑Server) Model

Build a control server that interacts with Redis; clients only request a batch of tokens when their local bucket is depleted, reducing network overhead. This is similar to Java collection resizing and still incurs some error due to batch granularity.

6. Open‑Source Project

A practical implementation of the invoice‑server approach is the SnowJean project (https://github.com/yueshutong/SnowJena). It uses the observer pattern for dynamic rule configuration, the factory pattern for creating limiters, and the builder pattern for constructing rate‑limiting rules.

The project also monitors client health via Redis key expiration and heartbeat messages, and provides an ECharts‑based QPS dashboard.