Distributed Service Rate Limiting, Degradation, and Caching Strategies

When a product API experiences a sudden surge in traffic, such as after a popular item becomes trending, it is essential to protect the service using caching, rate limiting, and degradation strategies.

Service Rate Limiting

Rate limiting controls the request rate either by limiting concurrent accesses or by limiting the number of requests within a time window, rejecting, queuing, or degrading requests once the limit is reached.

Rate‑Limiting Algorithms

Leaky Bucket Algorithm – Requests are placed into a bucket; if the bucket is full, excess requests are dropped or handled by a fallback strategy. The bucket drains at a fixed rate, ensuring the outflow never exceeds the configured limit.

Token Bucket Algorithm – Tokens are added to a bucket at a steady rate (v = time period / limit). A request consumes a token; if none are available, the request is limited. This algorithm permits short bursts of traffic.

Sliding Window Algorithm – The time window is divided into smaller sub‑windows; each sub‑window records its request count. When the sum of counts across sub‑windows exceeds the threshold, limiting is triggered.

Message queues can be used to implement these algorithms.

Access‑Layer Rate Limiting

Nginx Rate Limiting – Nginx uses the leaky bucket algorithm via the ngx_http_limit_req_module (limit_req) to restrict request rates based on client attributes such as IP or User‑Agent.

Local Interface Rate Limiting

Java’s Semaphore from the concurrency library can control the number of simultaneous accesses to a resource.

private final Semaphore permit = new Semaphore(40, true);

public void process() {
    try {
        permit.acquire();
        // TODO: handle business logic
    } catch (InterruptedException e) {
        e.printStackTrace();
    } finally {
        permit.release();
    }
}

Distributed Interface Rate Limiting

Using a message queue (e.g., MQ middleware or Redis List) as a buffer, requests are queued when the traffic exceeds a threshold, and consumers process them at a rate matching service throughput, following the leaky bucket principle.

Service Degradation

When traffic spikes despite prior risk control, a fallback plan can be activated to degrade non‑critical services, either by delaying or pausing them.

Degradation Strategies

Stop Edge Services – Disable low‑priority features (e.g., historical order queries during a sales event) to preserve core service availability.

Reject Requests – When request volume exceeds the limit or failures increase, reject a portion of incoming traffic.

Reject Policies – Random rejection, reject oldest requests, or reject non‑core requests based on a predefined critical‑service list.

Recovery Plan

After degradation, additional consumer services can be registered to handle the remaining load, and slow‑loading mechanisms can be applied to balance the system.

Data Caching

With risk control in place, the following steps can be taken during a traffic surge:

Use a distributed lock to block concurrent modifications.

Cache hot data in a caching middleware.

Allow requests to read/write the cache first.

Send the final results to a message queue for asynchronous consumption.

Cache Issues

Consider an inventory service with 100 items. If all requests read from a cached value, the cache can become a bottleneck.

Read‑Write Separation – Deploy Redis Sentinel with master‑slave replication; reads are served by slaves while writes go to the master, allowing fast failure when inventory reaches zero.

Load Balancing – Split the inventory across multiple cache nodes (e.g., 10 items per node) and distribute requests evenly, similar to the design of ConcurrentHashMap ’s counterCells .

Page Cache – Aggregate short‑term write operations in a cache before flushing them to the underlying storage, a technique used in operating systems and databases.

Specific implementation details can be found in related articles.

Overall, combining rate limiting, degradation, and intelligent caching helps maintain service stability during sudden traffic spikes in distributed e‑commerce systems.