How to Safeguard Microservices with Smart Rate‑Limiting Strategies

Overview

Service rate limiting protects a system by rejecting excess requests when traffic exceeds capacity, preventing resource exhaustion and crashes.

Problem Domain

High‑traffic spikes such as ticket‑booking flash sales generate massive concurrent reads and writes, bot traffic, and often lack caching or queueing. The result is slow responses or total failure. Rate limiting provides fast‑fail behavior and informs callers when capacity is exhausted.

Capability Model

Three capabilities are typically required:

Global rate limiting : a single QPS ceiling for the whole service, independent of request source.

Tag‑based rate limiting : classify requests by tags (user, API, region, etc.) and apply fine‑grained limits.

Dynamic adjustment : each instance adjusts its quota in real time based on observed metrics.

Rate‑Limiting Algorithms

Common algorithms and their trade‑offs:

Fixed window – simple to implement; suffers from traffic bursts at window boundaries.

Sliding window – smooths bursts; provides no buffering when the limit is exceeded.

Leaky bucket – steady output rate with limited buffering; cannot absorb instantaneous spikes.

Token bucket – combines buffering and burst handling; requires careful tuning of the initial token count.

TSF Distributed Rate‑Limiting Architecture

TSF (Tencent Service Framework) uses a token‑bucket algorithm. A central limit‑control component calculates per‑instance QPS quotas and pushes them to each instance. Instances report request statistics via the TSF‑SDK; the control component continuously refines quotas. Requests that exceed the quota receive HTTP 429 (Too Many Requests).

Configuration Styles

TSF supports two rule types:

Global limit : a single QPS ceiling applied to all requests of a service.

Tag limit : rules that match request tags (e.g., API name, user group) for more precise control.

Multiple rules may coexist; a request is allowed only if it passes every applicable rule.

Integration with Elastic Scaling

TSF auto‑scaling monitors QPS, latency, CPU and memory across instances. When thresholds are crossed, the system adds or removes instances. Because scaling decisions take at least a minute while rate limiting reacts within seconds, the two mechanisms must be coordinated to avoid over‑provisioning or premature throttling.

Testing Guidelines

Before deploying limits, perform load testing that mirrors production hardware. Recommended practices:

Keep the test environment identical to production (CPU, memory, network).

Vary a single factor at a time (instance count, CPU, network bandwidth, etc.).

Prioritize core APIs in the test plan.

Observe QPS plateau, error‑rate spikes, and latency growth as load increases.

Use mock services to isolate the capacity of individual components before full‑chain testing.

Typical test topologies:

Single‑instance service.

Multi‑instance service behind a load balancer.

Gateway‑only.

Gateway + service chain.

Set limits so that safe QPS keeps CPU utilization around 70‑80 %.

E‑Commerce Flash‑Sale Example

During a large promotion (e.g., Double 11), a core service might be configured with a global limit of 1500 QPS and an additional tag‑based limit of 1000 QPS for a hotspot API. The layered limits protect the gateway and downstream services from overload.

Sample Mock Code for Load Testing

public String mockResponseTime() throws InterruptedException {
    int responseTime = 0;
    Random randomProportion = new Random();
    Random randomOffset = new Random();
    int proportion = randomProportion.nextInt(100);
    int offset = randomOffset.nextInt(5);
    if (proportion < 80) {
        responseTime = calcTime(50, offset);
    } else if (proportion >= 80 && proportion < 95) {
        responseTime = calcTime(200, offset);
    } else {
        responseTime = calcTime(400, offset);
    }
    Thread.sleep(responseTime);
    return "OK";
}
private int calcTime(int milliSecond, int offset) {
    Random random = new Random();
    return (random.nextInt(100) % 2 == 0) ? milliSecond + offset : milliSecond - offset;
}

References

https://cloud.tencent.com/document/product/649/30719

https://cloud.tencent.com/document/product/649/19046