Practical Guide to Rate Limiting: Algorithms, Implementation, and Production Cases

Preface

Last summer, I received an alarm from a financial platform: the payment interface error rate surged to 35%. When I arrived at the data center, I found the database connection pool exhausted and a massive backlog of requests—an obvious case of missing rate‑limiting protection. It is like a highway without toll gates; during peak hours it becomes a parking lot.

The essence of rate limiting is not to reject service outright, but to sacrifice controllable traffic to protect the core link.

During a major e‑commerce promotion, a token‑bucket algorithm limited the flash‑sale API QPS to 50,000, losing 20% of burst traffic but preserving 99% of core transaction success rate.

1 Common Rate‑Limiting Schemes

1.1 Fixed‑Window Counter

Core principle: Count requests within a fixed time window (e.g., 1 second). If the count exceeds the threshold, subsequent requests are rejected.

Specific code implementation:

// Thread‑safe implementation (AtomicLong optimized version)
public class FixedWindowCounter {
    private final AtomicLong counter = new AtomicLong(0);
    private volatile long windowStart = System.currentTimeMillis();
    private final int maxRequests;
    private final long windowMillis;

    public boolean tryAcquire() {
        long now = System.currentTimeMillis();
        if (now - windowStart > windowMillis) {
            if (counter.compareAndSet(counter.get(), 0)) {
                windowStart = now;
            }
        }
        return counter.incrementAndGet() <= maxRequests;
    }
}

Fatal flaw: If a limit of 100 requests per second is set, a burst of 100 requests at 0.9 s and another 100 at 0.1 s of the next second will allow 200 requests in two seconds, creating a “critical‑point spike”.

Applicable scenarios: Log collection, coarse‑grained throttling of non‑critical interfaces.

1.2 Sliding Window

Core principle: Divide the time window into smaller slices (e.g., 10 s) and count the total requests of the most recent N slices.

Redis‑based Lua script implementation:

// Redis Lua implementation of a sliding window (millisecond precision)
String lua = "
    local now = tonumber(ARGV[1])
    local window = tonumber(ARGV[2])
    local key = KEYS[1]

    redis.call('ZREMRANGEBYSCORE', key, '-inf', now - window)
    local count = redis.call('ZCARD', key)

    if count < tonumber(ARGV[3]) then
        redis.call('ZADD', key, now, now)
        redis.call('EXPIRE', key, window/1000)
        return 1
    end
    return 0
";

Technical highlight: A securities trading system reduced API exception rate from 5% to 0.3% after adopting sliding windows, achieving ±10 ms time‑slice accuracy via Redis ZSET.

Advantages comparison

Metric

Fixed Window

Sliding Window

Time precision

1 s

100 ms

Critical‑point spike

Exists

Eliminated

Implementation complexity

Simple

Medium

2.3 Leaky Bucket Algorithm

Core principle: Requests flow into a bucket like water; the system processes them at a fixed rate. When the bucket is full, new requests are dropped.

Specific implementation:

// Leaky bucket dynamic implementation (Semaphore optimized version)
public class LeakyBucket {
    private final Semaphore permits;
    private final ScheduledExecutorService scheduler;

    public LeakyBucket(int rate) {
        this.permits = new Semaphore(rate);
        this.scheduler = Executors.newScheduledThreadPool(1);
        scheduler.scheduleAtFixedRate(() -> permits.release(rate), 1, 1, TimeUnit.SECONDS);
    }

    public boolean tryAcquire() {
        return permits.tryAcquire();
    }
}

Technical pain point: An IoT platform using this scheme could handle 100 k devices reporting simultaneously at a stable 500 req/s, but burst traffic may cause queue buildup, similar to a milk‑tea funnel where pearls get stuck.

Applicable scenarios: IoT command dispatch, payment channel quota enforcement where a strict constant rate is required.

1.4 Token Bucket Algorithm

Core principle: Tokens are generated at a fixed rate; a request must acquire a token before execution. Bursts can consume accumulated tokens.

Specific implementation using Guava RateLimiter:

// Guava RateLimiter advanced usage
RateLimiter limiter = RateLimiter.create(10.0, 1, TimeUnit.SECONDS); // warm‑up
limiter.acquire(5); // try to get 5 tokens
// Dynamic rate adjustment (requires reflection)
Field field = RateLimiter.class.getDeclaredField("tokens");
field.setAccessible(true);
AtomicDouble tokens = (AtomicDouble) field.get(limiter);
tokens.set(20); // inject 20 tokens during a burst

Real‑world case: A video platform limited normal traffic to 100 k QPS, but during hot events allowed a 50% over‑limit for three seconds, preventing avalanche while preserving user experience.

Dynamic features:

Normal state: limit QPS.

Burst: allow temporary over‑draw.

Sustained burst: tokens are exhausted, limiting further traffic.

2 Production‑Level Practices

2.1 Distributed Rate Limiting at the Gateway Layer

An e‑commerce Double‑11 solution used Redis + Lua for distributed counting together with Nginx local cache, blocking 83% of malicious requests at the gateway.

2.2 Adaptive Circuit‑Breaker Mechanism

A social platform automatically lowered the rate‑limit threshold from 50 k to 30 k during traffic spikes and gradually restored it after recovery.

3 Pitfalls and Performance Optimization

3.1 Fatal Pitfalls

Applying rate limiting before the database connection pool can cause connection leaks and crash the database.

Correct practice follows the three principles of circuit breaking:

Fast failure: reject invalid requests at the entry layer.

Dynamic downgrade: keep core services with minimal resources.

Automatic recovery: gradually ramp up after a break.

3.2 Performance Optimization

A financial system’s JMH benchmark showed that replacing AtomicLong with LongAdder increased rate‑limiting throughput by 220%.

Optimization techniques include reducing CAS contention and using segmented lock bases.

Conclusion

The article listed the four most commonly used rate‑limiting schemes in practice.

Different business scenarios require selecting the appropriate rate‑limiting strategy.

The golden rule of rate limiting is akin to a high‑speed rail gate: it ensures traffic efficiency while safeguarding safety.