Mastering Rate Limiting: Algorithms, Strategies, and Real-World Implementations

Why Rate Limiting?

In daily life and online services, limiting traffic prevents overload and ensures a good experience. For example, a scenic spot caps visitors at 10,000; when 30,000 arrive, limiting entry avoids accidents. Similarly, a website must throttle spikes to keep servers available.

Rate Limiting Strategies

Circuit Breaker

Design systems to open a circuit when problems persist, rejecting traffic until recovery. Tools like Hystrix and Sentinel implement this.

Service Degradation

When traffic surges, downgrade non‑critical features (e.g., comments, points) to free resources for core functions. Degradation can also be achieved by reading/writing only caches.

Delay Processing

Buffer incoming requests in a queue and process them sequentially, reducing immediate load but may introduce latency.

Privilege Handling

Classify users and give higher‑priority groups preferential access while delaying or rejecting others.

Rate Limiting Algorithms

Counter Algorithm

Simple counting, e.g., allow at most 100 requests per minute by incrementing a counter and resetting after the time window.

Leaky Bucket Algorithm

Requests enter a bucket that leaks at a constant rate; excess requests overflow, providing peak‑shaping and buffering.

Token Bucket Algorithm

Tokens are added at a steady rate; a request consumes a token, allowing bursts while respecting the average rate.

Concurrent Limiting

Set overall QPS limits, configure server parameters (e.g., Tomcat's maxThreads, nginx limit_conn), and limit database or thread pool connections.

Interface Limiting

Control total calls per interval and use sliding windows for finer‑grained rate control.

Implementation Examples

Guava

LoadingCache<Long, AtomicLong> counter = CacheBuilder.newBuilder()
    .expireAfterWrite(2, TimeUnit.SECONDS)
    .build(new CacheLoader<Long, AtomicLong>() {
        @Override
        public AtomicLong load(Long second) {
            return new AtomicLong(0);
        }
    });
counter.get(1L).incrementAndGet();

Token Bucket with Guava RateLimiter

RateLimiter limiter = RateLimiter.create(2);
System.out.println(limiter.acquire());
// sleep 2000 ms
System.out.println(limiter.acquire());
// more acquires...

Distributed Limiting with Nginx + Lua

local locks = require "resty.lock"
local function acquire()
    local lock = locks:new("locks")
    local elapsed, err = lock:lock("limit_key")
    local limit_counter = ngx.shared.limit_counter
    local key = "ip:" .. os.time()
    local limit = 5
    local current = limit_counter:get(key)
    if current ~= nil and current + 1 > limit then
        lock:unlock()
        return 0
    end
    if current == nil then
        limit_counter:set(key, 1, 1)
    else
        limit_counter:incr(key, 1)
    end
    lock:unlock()
    return 1
end
ngx.print(acquire())