Mastering Rate Limiting: When to Use Fixed, Sliding, Leaky or Token Buckets

How to Implement Rate Limiting

Before diving into algorithms, understand that circuit breaking prevents total system collapse, while frequent circuit trips degrade user experience; rate limiting keeps traffic within the system’s processing capacity, ensuring stability and better resource utilization.

Step 1 – Determine System Capacity

Perform pressure testing (load testing) in an isolated environment or on a production node to obtain two key metrics: request rate (e.g., requests per second) and concurrency (maximum simultaneous requests). These numbers become the basis for setting rate‑limit thresholds.

Step 2 – Define Traffic‑Intervention Strategy

Four common strategies can be grouped as “two windows, two buckets”.

Fixed Window

A fixed window defines a constant time slice (e.g., 1 minute, 30 seconds). Requests are counted within each slice; when the count exceeds the threshold, traffic is rejected until the next slice starts.

Simple to implement, but if requests arrive in bursts the fixed threshold may either over‑limit or under‑utilize resources.

int totalCount = 0; // global counter, reset by a timer each fixed period if (totalCount > limit) { return; // reject request } totalCount++; // do something...

When the request burst is very concentrated, the fixed‑window threshold effectively becomes the maximum concurrent load, so a shorter window is advisable.

Sliding Window

A sliding window refines the fixed window by dividing the period into finer sub‑windows (e.g., 60 one‑second slots for a 1‑minute window) and moving the counting range forward with time, smoothing out bursts.

If the fixed window is already very small (e.g., 1 second), a sliding window adds little value and may increase overhead.

List<int>[] counterList = new List<int>[windowCount]; int sum = counterList.Sum(); if (sum > limit) { return; // reject request } int currentIndex = currentSecond % windowCount; counterList[currentIndex]++; // do something...

Leaky Bucket

The leaky‑bucket algorithm fixes the output rate. Incoming requests are queued in a buffer; the bucket “leaks” at a constant rate. If the buffer overflows, requests are dropped.

int unitSpeed = 0; // current output rate int waterLevel = 0; // buffer level if (unitSpeed < speedThreshold) { unitSpeed++; // do something... } else { if (waterLevel > waterThreshold) { return; // drop request } waterLevel++; while (unitSpeed >= speedThreshold) { sleep(shortTime); } unitSpeed++; waterLevel--; // do something... }

This approach balances protection and flexibility, making it a good general‑purpose solution.

Token Bucket

The token‑bucket algorithm fixes the input rate. Tokens are generated at a steady pace and stored in a bucket; a request proceeds only if a token is available.

int tokenCount = tokenThreshold; // available tokens if (tokenCount == 0) { return; // reject request } tokenCount--; // do something...

When traffic volume is high and token generation exceeds processing speed, the system can achieve near‑maximum throughput.

Best‑Practice Recommendations

Use fixed window only for quick, temporary mitigation.

Choose sliding window when you can tolerate occasional spikes and need simple implementation.

Prefer leaky bucket as a versatile default; it offers “wide‑in, narrow‑out” protection.

Adopt token bucket for maximum performance when traffic is relatively stable and you can provision a bucket size larger than the peak concurrency.

Challenges in Distributed Systems

Vertical (front‑line) considerations : Apply rate limiting at the ingress layer (e.g., Nginx’s ngx_http_limit_conn_module or ngx_http_limit_req_module) if available. Otherwise, implement AOP‑style limits in the application layer, targeting high‑frequency services first.

Client‑side vs. server‑side : Client‑side limiting reduces connection overhead and spreads load, but requires coordination across nodes. Server‑side limiting is cheaper to implement but concentrates pressure on the server.

Horizontal (across nodes) considerations : Nodes have varying performance; use a monitoring platform and a configuration center to propagate threshold changes quickly across the cluster.

Conclusion

Rate limiting acts like a fuse: once the predefined threshold is reached, traffic is cut off to protect the system. Combined with circuit breaking and appropriate degradation strategies, it forms a robust defense against overload in modern distributed back‑end architectures.