Implementing Dynamic Rate Limiting with Caffeine Cache in Java

Previously a simple rate‑limiting solution was built using JDK's Map, but it lacked a clean asynchronous reset mechanism. This article introduces a high‑performance local cache Caffeine to handle dynamic rate limiting based on TPS, using a 1‑second refresh interval.

The design uses two data structures: a Map storing each request's TPS configuration and a Caffeine cache holding an AtomicInteger counter for each request identifier.

Data Structure Selection : Map for configuration and Caffeine cache for counters.

Rate‑limit Decision Logic : Retrieve the counter from the cache, compare it with the configured TPS, return true if the limit is exceeded, otherwise increment the counter and return false.

Dynamic Configuration : Add or update request identifiers and their TPS values at runtime.

Using Caffeine Cache : Configure the cache to expire entries after 1 second, ensuring counters are refreshed automatically.

Implementation code:

import com.github.benmanes.caffeine.cache.Caffeine
import com.github.benmanes.caffeine.cache.LoadingCache

import java.util.concurrent.TimeUnit
import java.util.concurrent.atomic.AtomicInteger

/** Rate‑limit tool based on Caffeine, supports dynamic configuration by TPS */
class TpsLimit {
    Map<String, Integer> qpsConfig = [:]

    LoadingCache<Object, AtomicInteger> build = Caffeine.newBuilder()
        .refreshAfterWrite(1, TimeUnit.SECONDS)
        .build(key -> new AtomicInteger())

    /** Whether to limit */
    boolean isLimit(String key) {
        AtomicInteger atomicInteger = build.get(key)
        if (atomicInteger.get() >= qpsConfig.get(key, 1)) {
            return true
        }
        atomicInteger.incrementAndGet()
        return false
    }

    /** Add rate‑limit configuration */
    def addConfig(String key, Integer qps) {
        qpsConfig.put(key, qps)
    }
}

Test script demonstrates a 1 TPS configuration, repeatedly sending requests and printing “未限流” (not limited) when the limit is not reached.

import com.funtester.httpclient.FunHttp
import com.funtester.utils.TpsLimit

class Routine extends FunHttp {
    static void main(String[] args) {
        def limit = new TpsLimit()
        limit.addConfig("test", 1)
        1000.times {
            sleep(0.1)
            fun {
                def limit1 = limit.isLimit("t4est")
                if (!limit1) {
                    output("未限流")
                }
            }
        }
    }
}

Console output shows several “未限流” lines, confirming the default 1 TPS setting works as expected.