Caching, Rate Limiting, Smoothing, and Idempotency: Solving Concurrency Problems

Handling high concurrency is not only a technical challenge but also a way for architects to turn business uncertainty into deterministic system capability. Common techniques include caching, rate limiting, smoothing (peak shaving), and idempotency, each addressing a specific problem.

Cache

Cache aims to reduce repeated access to slow resources such as databases, disks, or third‑party APIs. The article uses a milk‑tea shop analogy: the recipe for the popular pearl milk tea never changes, yet staff keep flipping through the recipe book for each order, wasting time. Caching is like posting the recipe on the wall so anyone can read it instantly. In a system, the database is the recipe book; frequently requested but rarely changed data (e.g., product details, user profiles, hot lists) would otherwise hammer the database. A cache such as Redis stores these immutable or infrequently changed items in memory, delivering them more than 100× faster.

Rate Limiting

Rate limiting protects a system from traffic that exceeds its capacity. In the shop scenario, the maximum throughput is 100 cups per minute, but 1,000 orders arrive. The two choices are to push through and collapse the operation, or to limit intake to 100 per minute and ask the rest to retry later. Rate limiting adopts the latter: it prefers partial service over total failure, keeping the system alive for later requests. Common algorithms include:

Fixed window : simple, but suffers boundary spikes (e.g., 100 requests at second 59 and another 100 at second 61 cause 200 requests in two seconds).

Sliding window : smooths statistics to avoid the boundary issue.

Token bucket : tokens are added to a bucket at a steady rate; a request proceeds only if a token is available, allowing limited bursts.

Leaky bucket : incoming requests fill a bucket that drains at a fixed rate; excess requests are dropped, providing strict smoothing without bursts.

Smoothing (Peak Shaving)

Smoothing, also called peak shaving, buffers sudden traffic spikes using a message queue. Unlike rate limiting, which rejects excess requests, smoothing delays them. The shop example shows 10,000 customers arriving at noon, but only 100 arriving at 12:30; the issue is the instantaneous concentration, not total volume. The solution is to store the burst in a queue—like a reservoir that captures a sudden rainstorm and releases water gradually downstream. In a flash‑sale scenario, 1 million “buy now” clicks at 12:00 are enqueued; the backend consumes 10 k messages per second, finishing in 100 seconds while maintaining stability. The key distinction: rate limiting = reject, smoothing = delay.

Idempotency

Idempotency ensures that the same operation, no matter how many times it is executed, yields the same result. This is crucial when networks are unreliable and retries occur. The article illustrates with a mobile payment: a stalled network leads a user to tap the payment button three times, potentially charging three times if the system lacks idempotency. Idempotency works by assigning a unique request ID (e.g., a UUID) on the client side. The server checks whether this ID has already been processed; if so, it returns the previous result, otherwise it performs the operation and records the ID. This is analogous to a take‑away order number: presenting the same number three times results in only one cup being handed out.

These four techniques are complementary rather than mutually exclusive. Real‑world high‑concurrency systems typically combine them: cache handles the bulk of read traffic, rate limiting guards the system’s upper bound, smoothing evens out write‑burst spikes, and idempotency guarantees data correctness. Omitting any of them can cause failures under heavy load.