Implementing Dynamic Rate Limiting with Caffeine Cache in Java

Background

The initial rate‑limiting implementation stored request counts in a plain Map and relied on an asynchronous daemon thread to reset the counters according to a configuration. Managing such daemon threads in ordinary Java projects is error‑prone and adds unnecessary complexity.

Solution Overview

This article replaces the thread‑based approach with Caffeine , a high‑performance local cache. Each request identifier (key) has a counter stored in a Caffeine cache entry that automatically expires after a configurable time window (here 1 second). The allowed transactions‑per‑second (TPS) for each key is kept in a separate Map, enabling dynamic updates without restarting the application.

Key Components

Data structures – A Map<String, Integer> holds the TPS limit for each key. A Caffeine LoadingCache<Object, AtomicInteger> stores a mutable AtomicInteger counter for the same key.

Rate‑limit decision logic – When a request arrives, the cache returns the current counter. If the counter is greater than or equal to the configured TPS, isLimit returns true (the request is throttled). Otherwise the counter is incremented and false is returned.

Dynamic configuration – New keys and their TPS limits can be added or updated at runtime via addConfig, allowing on‑the‑fly adjustments.

Cache expiration – The cache is built with refreshAfterWrite(1, TimeUnit.SECONDS). After one second the entry is refreshed (a new AtomicInteger is created), which effectively resets the request count and prevents stale data from occupying memory.

Implementation

import com.github.benmanes.caffeine.cache.Caffeine;
import com.github.benmanes.caffeine.cache.LoadingCache;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

/**
 * Rate‑limit utility based on Caffeine. Supports dynamic TPS configuration.
 */
class TpsLimit {
    // TPS configuration: key -> allowed requests per second
    Map<String, Integer> qpsConfig = [:];

    // Cache that holds a counter for each key; refreshed every second
    LoadingCache<Object, AtomicInteger> cache = Caffeine.newBuilder()
        .refreshAfterWrite(1, TimeUnit.SECONDS)
        .build(key -> new AtomicInteger());

    /**
     * Returns true if the request identified by {@code key} should be limited.
     */
    boolean isLimit(String key) {
        AtomicInteger counter = cache.get(key);
        int limit = qpsConfig.getOrDefault(key, 1);
        if (counter.get() >= limit) {
            return true; // limit exceeded
        }
        counter.incrementAndGet();
        return false; // within limit
    }

    /**
     * Adds or updates the TPS limit for a given key.
     */
    void addConfig(String key, Integer tps) {
        qpsConfig.put(key, tps);
    }
}

Usage Example (Test Script)

import com.funtester.httpclient.FunHttp;
import com.funtester.utils.TpsLimit;

class Routine extends FunHttp {
    static void main(String[] args) {
        TpsLimit limit = new TpsLimit();
        limit.addConfig("test", 1); // 1 request per second allowed
        1000.times {
            sleep(0.1); // 100 ms between attempts
            fun {
                boolean limited = limit.isLimit("test");
                if (!limited) {
                    output("未限流"); // "not limited"
                }
            }
        }
    }
}

Observed Result

Running the script prints lines such as:

22:19:20:545  未限流
22:19:20:644  未限流
22:19:22:094  未限流
22:19:22:195  未限流
22:19:24:048  未限流
22:19:24:150  未限流
22:19:25 uptime:6 s
22:19:25 finished: 49 task

Because the cache refreshes every second, the counter is reset after each interval. With a default configuration of 1 TPS, most requests pass the limit check, demonstrating that the Caffeine‑based solution provides lightweight, automatically‑resetting rate limiting without explicit background threads.