Mastering Rate Limiting: 4 Proven Strategies to Boost System Resilience

Introduction

Last summer, a financial platform reported a payment‑interface error rate soaring to 35%. When I arrived at the data‑center, the database connection pool was exhausted and requests piled up—a classic case of missing rate‑limiting protection.

Rate limiting is not about refusing service; it is about sacrificing controllable traffic to protect core pathways.

During a major e‑commerce promotion, a token‑bucket algorithm limited the flash‑sale API to 50 k QPS, losing 20% of burst traffic but preserving 99% of core transaction success.

1 Common Rate‑Limiting Solutions

1.1 Fixed Window Counter

Core principle: Use a fixed time window (e.g., 1 second) to count requests; if the count exceeds the threshold, reject subsequent requests.

Specific code implementation:

// Thread‑safe implementation (AtomicLong optimized version)
public class FixedWindowCounter {
    private final AtomicLong counter = new AtomicLong(0);
    private volatile long windowStart = System.currentTimeMillis();
    private final int maxRequests;
    private final long windowMillis;

    public boolean tryAcquire() {
        long now = System.currentTimeMillis();
        if (now - windowStart > windowMillis) {
            if (counter.compareAndSet(counter.get(), 0)) {
                windowStart = now;
            }
        }
        return counter.incrementAndGet() <= maxRequests;
    }
}

Critical flaw: If the limit is 100 requests per second, a burst of 100 requests at 0.9 s and another 100 at 0.1 s of the next second results in 200 requests passing in two seconds.

Analogy: like cars racing the traffic light when it turns green, causing a “critical spike”.

Applicable scenarios: Log collection, coarse‑grained throttling of non‑critical APIs.

1.2 Sliding Window

Core principle: Subdivide the time window into smaller slices (e.g., 10 seconds) and count the total requests in the most recent N slices.

Redis Lua script implementation (millisecond precision):

// Redis Lua implementation of sliding window (millisecond precision)
String lua = "
    local now = tonumber(ARGV[1])
    local window = tonumber(ARGV[2])
    local key = KEYS[1]

    redis.call('ZREMRANGEBYSCORE', key, '-inf', now - window)
    local count = redis.call('ZCARD', key)

    if count < tonumber(ARGV[3]) then
        redis.call('ZADD', key, now, now)
        redis.call('EXPIRE', key, window/1000)
        return 1
    end
    return 0
";

Technical highlight: A securities‑trading system reduced API error rate from 5% to 0.3% after adopting the sliding‑window algorithm.

Implementation uses Redis ZSET for time slices, keeping error within ±10 ms.

1.3 Leaky Bucket Algorithm

Core principle: Requests enter a bucket like water; the system processes them at a fixed rate. When the bucket is full, new requests are dropped.

Specific implementation:

// Leaky bucket dynamic implementation (Semaphore optimized version)
public class LeakyBucket {
    private final Semaphore permits;
    private final ScheduledExecutorService scheduler;

    public LeakyBucket(int rate) {
        this.permits = new Semaphore(rate);
        this.scheduler = Executors.newScheduledThreadPool(1);
        scheduler.scheduleAtFixedRate(() -> permits.release(rate), 1, 1, TimeUnit.SECONDS);
    }

    public boolean tryAcquire() {
        return permits.tryAcquire();
    }
}

Technical pain point: An IoT platform used this scheme to handle 100 k devices reporting simultaneously, processing at a stable 500 records/second, but burst traffic caused queue buildup, similar to a clogged funnel.

Applicable scenarios: IoT command dispatch, payment‑channel quota enforcement where a constant processing rate is required.

1.4 Token Bucket Algorithm

Core principle: Tokens are generated at a fixed rate; a request must acquire a token before execution. Bursts can consume accumulated tokens.

Implementation using Guava RateLimiter:

// Guava RateLimiter advanced usage
RateLimiter limiter = RateLimiter.create(10.0, 1, TimeUnit.SECONDS); // warm‑up
limiter.acquire(5); // try to get 5 tokens

// Dynamically adjust rate via reflection (for illustration only)
Field field = RateLimiter.class.getDeclaredField("tokens");
field.setAccessible(true);
AtomicDouble tokens = (AtomicDouble) field.get(limiter);
tokens.set(20); // inject 20 tokens during a spike

Real‑world case: A video platform limited normal traffic to 100 k QPS but allowed a 50% over‑limit for three seconds during hot events, preventing avalanche failures while preserving user experience.

Dynamic characteristics:

Normal operation: limit QPS.

Burst: allow token over‑draw.

Sustained burst: tokens deplete, limiting resumes.

2 Production‑Level Practices

2.1 Distributed Rate Limiting at the Gateway

An e‑commerce Double‑11 solution used Redis + Lua for distributed counting combined with Nginx local cache, blocking 83% of malicious requests at the gateway.

2.2 Adaptive Circuit‑Breaker Mechanism

A social platform automatically lowered the rate‑limit threshold from 50 k to 30 k during traffic spikes and gradually restored it after recovery.

3 Pitfalls and Performance Optimizations

3.1 Fatal Mistakes

Applying rate limiting **before** the database connection pool can cause connection leaks and crash the database.

Correct approach follows the three principles of circuit breaking:

Fast failure: reject invalid requests at the entry point.

Dynamic downgrade: keep core services with minimal resources.

Automatic recovery: gradually ramp up after a circuit break.

3.2 Performance Tuning

A financial system measured with JMH found that replacing AtomicLong with LongAdder increased rate‑limiting throughput by 220%.

Optimization techniques include reducing CAS contention and using segmented locks.

Conclusion

The four most common rate‑limiting schemes—fixed window, sliding window, leaky bucket, and token bucket—cover a wide range of scenarios. Selecting the appropriate algorithm depends on business requirements and traffic characteristics.

Remember: a good rate‑limiting solution is like a high‑speed rail gate—ensuring smooth flow while safeguarding the safety line.