Mastering Rate Limiting: Algorithms, Strategies, and Real‑World Implementations

In everyday scenarios such as crowded tourist attractions, limiting the number of visitors ensures safety and a good experience; the same principle applies to online services that experience traffic spikes, like celebrity news or flash sales, where uncontrolled traffic can cause system crashes.

Rate‑Limiting Concepts

The goal of rate limiting is to keep a service usable by allowing only a controlled amount of traffic while queuing or rejecting excess requests.

Common Strategies

Circuit Breaker : Detects failures and automatically opens a switch to reject traffic, preventing overload. When the backend recovers, the circuit closes.

Service Degradation : Downgrades or disables non‑critical features (e.g., comments, points) during spikes to free resources for core functions.

Delay Processing : Buffers incoming requests in a queue (e.g., a leaky‑bucket) and processes them later, reducing immediate load at the cost of latency.

Privilege Processing : Prioritizes high‑value users, serving them first while delaying or rejecting others.

Difference Between Cache, Degrade, and Limit

Cache : Increases throughput and speeds up access.

Degrade : Provides fallback responses when components fail or resources are exhausted.

Limit : Enforces thresholds when caching and degradation are insufficient, protecting the service before it becomes unavailable.

Rate‑Limiting Algorithms

Three main families are used:

1. Counter Algorithm

Simple counting of requests within a fixed window (e.g., no more than 100 calls per minute). When the counter exceeds the limit, further requests are rejected. The counter resets after the window expires.

2. Leaky‑Bucket Algorithm

Requests enter a bucket; the bucket drains at a constant rate. If the incoming rate exceeds the drain rate, excess requests overflow and are dropped, smoothing traffic bursts.

3. Token‑Bucket Algorithm

Tokens are added to a bucket at a steady rate. A request consumes a token; if none are available, the request is rejected. This allows short bursts while enforcing an average rate.

Concurrency Limiting

Typical settings include total concurrent connections (e.g., database pools, thread pools), instantaneous connections (e.g., Nginx limit_conn), and average rate within a time window (e.g., Guava RateLimiter, Nginx limit_req).

Interface Limiting

Two aspects are covered: total call count within a period (counter algorithm) and sliding‑window algorithms that divide the period into finer slots for more precise control.

Implementation Examples

Guava RateLimiter (Java)

<dependency>
  <groupId>com.google.guava</groupId>
  <artifactId>guava</artifactId>
  <version>28.1-jre</version>
</dependency>

LoadingCache<Long, AtomicLong> counter = CacheBuilder.newBuilder()
    .expireAfterWrite(2, TimeUnit.SECONDS)
    .build(new CacheLoader<Long, AtomicLong>() {
        @Override
        public AtomicLong load(Long second) throws Exception {
            return new AtomicLong(0);
        }
    });
counter.get(1L).incrementAndGet();

Token Bucket with Guava RateLimiter

RateLimiter limiter = RateLimiter.create(2); // 2 tokens per second
System.out.println(limiter.acquire()); // blocks until a token is available
Thread.sleep(2000);
System.out.println(limiter.acquire());

Distributed Limiting with Nginx + Lua

Using resty.lock for atomic operations and lua_shared_dict for counters.

local locks = require "resty.lock"
local function acquire()
  local lock = locks:new("locks")
  local elapsed, err = lock:lock("limit_key")
  local limit_counter = ngx.shared.limit_counter
  local key = "ip:" .. os.time()
  local limit = 5
  local current = limit_counter:get(key)
  if current ~= nil and current + 1 > limit then
    lock:unlock()
    return 0
  end
  if current == nil then
    limit_counter:set(key, 1, 1) -- expires in 1 second
  else
    limit_counter:incr(key, 1)
  end
  lock:unlock()
  return 1
end
ngx.print(acquire())

These examples demonstrate how to apply rate‑limiting techniques in various environments, from single‑node Java services to distributed Nginx‑Lua setups.