1 Million QPS Coupon‑Grab System: Distributed Rate Limiting, Stock Capping, and CAP Trade‑offs

Problem Essence

During large promotions many teams treat a "coupon‑grab" feature as a simple high‑concurrency endpoint (click → eligibility check → stock decrement → success). In production the system often fails because boundary control is unclear, leading to stock over‑issue, DB connection exhaustion, message backlog, and massive time‑outs.

The core goal is to allocate limited resources to the most valuable requests under extreme load while preventing a single component failure from causing a full system avalanche.

Four Iron Rules for Production‑Grade Coupon Systems

Rate limiting is part of the business admission system, not an isolated component.

Stock capping, eligibility checks, idempotency, degradation, and compensation must be co‑designed.

CAP trade‑offs favor availability with bounded consistency rather than absolute strong consistency.

Why Many Rate‑Limiting Designs Still Crash

Interface‑Level QPS Limiting Only

Example: /coupon/grab limited to 50 k QPS globally. This shares the same gate for hot‑selling and normal coupons, mixes new/old users, bots, and channels, causing a single channel anomaly to drag down the whole activity.

Correct approach: isolate limits by activity, coupon template, user, and channel dimensions.

Redis‑Only Limiting Without Local Fallback

Redis may stay up but become slow: hot slot CPU spikes, response time jumps from 1 ms to 150 ms, threads block, and the system mistakenly believes the limiter is still alive.

No Definition of "Successful Request Budget"

Instead of asking "how many requests can the system handle?", ask "how many qualified requests should be allowed into the core issuing pipeline within the activity window?" Example parameters: stock = 100 k, average processing = 20 ms, target throughput = 12 k TPS, qualification rate = 35 %.

CAP Trade‑offs in a Coupon System

Gateway coarse limiting – protect entry service, quickly reject abnormal traffic – AP

Application global limiting – control total traffic entering core – AP

User eligibility cache – fast decision whether to continue processing – AP

Final stock decrement – guarantee no over‑issue – CP

Issuing flow persistence – ensure traceable business result – CP

Async notification / audit – recoverable, compensatable – AP

The system is a layered mixed‑consistency architecture, not purely AP or CP.

Rate‑Limiting Model: From QPS to Budget‑Driven

Four Simultaneous Limiting Dimensions

System‑wide entry limiting

Per‑activity limiting

Per‑coupon‑template limiting (especially for high‑value coupons)

Per‑user frequency control

Budget Calculation Example

Given: stock = 100 k, activity duration = 15 s, core throughput = 12 k TPS, qualification rate = 35 %.

Theoretical demand: 100000 / 15 ≈ 6667 TPS Core capacity (12 k TPS) is sufficient.

Considering qualification and retry overhead, set total budget to 130000.

Per‑second release budget: 8000 ~ 10000 TPS.

Token Bucket Fits Coupon Bursts

Token bucket’s rate defines stable throughput, burst defines allowed instantaneous spikes, making it more suitable than fixed windows for flash‑sale traffic.

Core Implementation: Redis Distributed Token Bucket + Local Adaptive Degradation

Lua Script for Atomic Decision and Token Refill

-- KEYS[1] = token bucket key
-- ARGV[1] = capacity
-- ARGV[2] = refill_rate_per_sec
-- ARGV[3] = requested_tokens
-- return: {allowed(1/0), tokens_left, retry_after_ms}

local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local requested = tonumber(ARGV[3])

local now_resp = redis.call('TIME')
local now_ms = now_resp[1] * 1000 + math.floor(now_resp[2] / 1000)

local bucket = redis.call('HMGET', key, 'tokens', 'ts')
local last_tokens = tonumber(bucket[1])
local last_ts = tonumber(bucket[2])

if last_tokens == nil then
    last_tokens = capacity
    last_ts = now_ms
end

local delta = math.max(0, now_ms - last_ts)
local refill = delta * refill_rate / 1000
local current_tokens = math.min(capacity, last_tokens + refill)

local allowed = 0
local retry_after = 0
if current_tokens >= requested then
    allowed = 1
    current_tokens = current_tokens - requested
else
    local missing = requested - current_tokens
    retry_after = math.ceil(missing * 1000 / refill_rate)
end

redis.call('HMSET', key, 'tokens', current_tokens, 'ts', now_ms)
redis.call('PEXPIRE', key, math.ceil(capacity * 1000 / refill_rate) * 2)

return {allowed, math.floor(current_tokens), retry_after}

Key production details:

Use TIME to avoid clock drift across nodes.

Store state in a HASH to reduce overhead.

Set PEXPIRE to clean up cold keys after the activity.

Return retry_after_ms for front‑end feedback.

Java Wrapper Returning Rich Decision

public record RateLimitDecision(
        boolean allowed,
        String strategy,
        long tokensLeft,
        long retryAfterMs,
        String reason) {
    public static RateLimitDecision pass(String strategy, long tokensLeft) {
        return new RateLimitDecision(true, strategy, tokensLeft, 0L, "PASS");
    }
    public static RateLimitDecision reject(String strategy, long retryAfterMs, String reason) {
        return new RateLimitDecision(false, strategy, 0L, retryAfterMs, reason);
    }
}

@Service
public class RedisTokenBucketLimiter {
    private final RedissonClient redissonClient;
    private final RScript script;
    private final String luaScript;

    public RedisTokenBucketLimiter(RedissonClient redissonClient) {
        this.redissonClient = redissonClient;
        this.script = redissonClient.getScript(StringCodec.INSTANCE);
        this.luaScript = "..."; // same Lua as above
    }

    public RateLimitDecision tryAcquire(String key, int capacity, int refillRate, int requested) {
        List<Object> result = script.eval(RScript.Mode.READ_WRITE, luaScript,
                RScript.ReturnType.MULTI, List.of(key), capacity, refillRate, requested);
        long allowed = ((Number) result.get(0)).longValue();
        long tokensLeft = ((Number) result.get(1)).longValue();
        long retryAfterMs = ((Number) result.get(2)).longValue();
        if (allowed == 1L) {
            return RateLimitDecision.pass("REDIS_GLOBAL", tokensLeft);
        }
        return RateLimitDecision.reject("REDIS_GLOBAL", retryAfterMs, "GLOBAL_LIMITED");
    }
}

Hybrid Limiter: Switch to Local When Redis Slows

@Service
public class HybridCouponRateLimiter {
    private final RedisTokenBucketLimiter redisLimiter;
    private final LocalTokenBucketLimiter localLimiter;
    private final AtomicInteger slowCounter = new AtomicInteger();
    private final AtomicBoolean degraded = new AtomicBoolean(false);

    public RateLimitDecision acquire(String key, LimitProfile profile) {
        if (degraded.get()) {
            if (shouldProbe()) {
                return probeAndMaybeRecover(key, profile);
            }
            return localLimiter.tryAcquire(key, profile.localCapacity(), profile.localRefillRate());
        }
        long start = System.nanoTime();
        try {
            RateLimitDecision decision = redisLimiter.tryAcquire(key, profile.globalCapacity(),
                    profile.globalRefillRate(), 1);
            long costMs = (System.nanoTime() - start) / 1_000_000;
            onRedisLatency(costMs, profile);
            return decision;
        } catch (Exception ex) {
            degraded.set(true);
            return localLimiter.tryAcquire(key, profile.localCapacity(), profile.localRefillRate());
        }
    }

    private void onRedisLatency(long costMs, LimitProfile profile) {
        if (costMs >= profile.redisSlowThresholdMs()) {
            if (slowCounter.incrementAndGet() >= profile.degradeThreshold()) {
                degraded.set(true);
            }
        } else {
            slowCounter.updateAndGet(v -> Math.max(0, v - 1));
        }
    }

    private boolean shouldProbe() {
        return ThreadLocalRandom.current().nextInt(100) < 5;
    }

    private RateLimitDecision probeAndMaybeRecover(String key, LimitProfile profile) {
        try {
            RateLimitDecision probe = redisLimiter.tryAcquire("coupon:probe", 10, 10, 1);
            if (probe.allowed()) {
                degraded.set(false);
                slowCounter.set(0);
                return redisLimiter.tryAcquire(key, profile.globalCapacity(), profile.globalRefillRate(), 1);
            }
        } catch (Exception ignore) {}
        return localLimiter.tryAcquire(key, profile.localCapacity(), profile.localRefillRate());
    }
}

Local Limiter Must Adjust to Pod Count

If the local limit is a fixed "1000 QPS per pod", scaling the pod count changes the total capacity dramatically and breaks degradation control. The correct formula is:

local_limit_per_pod = global_fallback_limit / healthy_pod_count

Pod count can be obtained from service‑registry (Nacos/Eureka/Consul) or Kubernetes Endpoints.

Stock Capping Design: Three‑Stage Protection

Redis pre‑deduction : fast check to reject requests when stock is exhausted.

DB transaction confirmation : persist issuing record, decrement stock, write idempotent marker.

Asynchronous reconciliation : compare Redis pre‑deduction with DB result and compensate any discrepancy.

Idempotent key format (must contain activityId, couponTemplateId, userId, requestId):

public String buildIdempotentKey(Long activityId, Long templateId, Long userId, String requestId) {
    return "coupon:idem:%d:%d:%d:%s".formatted(activityId, templateId, userId, requestId);
}

Processing order before the core issuing transaction:

Redis/DB idempotency check.

User‑already‑claimed validation.

Eligibility verification.

Final stock transaction.

Real‑World Case: 1 Million Coupons, 3 s Peak 1.2 M QPS

Stock = 1 M, activity length = 60 s, peak = 1.2 M QPS, qualification rate = 25 %.

Gateway coarse limit: 250 k QPS to application layer, IP/device limit 2 req/s, blacklist immediate reject.

Application layer: separate token bucket for hot coupon (burst = 80 k, steady = 25 k TPS), per‑user 1 request per 3 s, pre‑filter already‑claimed users.

Core layer: Redis pre‑deduction → DB transaction → Kafka async success event; failures fail fast without long retries.

The design classifies traffic into three categories: obvious waste (blocked at gateway), potentially successful (compete for budget at application layer), and high‑value qualified requests (enter stock and transaction path).

Hot Key Sharding and Pitfalls

Using a single Redis key like coupon:activity:1001:template:8888 creates a hotspot slot. The solution is to shard the key, e.g.:

coupon:activity:1001:template:8888:slot:0
coupon:activity:1001:template:8888:slot:1
... 
coupon:activity:1001:template:8888:slot:31

Hash userId or deviceId to a fixed slot, distributing load while accepting a small bounded error.

Eligibility Cache Instead of DB Lookup

Pre‑warm a bitmap or Bloom filter in Redis before activity start.

Store whitelist in Redis or local cache for fast filtering.

Only a small fraction of cache misses fall back to asynchronous DB verification.

Engineering Enhancements

Hot‑Updateable Configuration

Parameters that must support runtime reload include global and template rate limits, local degradation thresholds, Redis slow‑call thresholds, channel quotas, and blacklist rules.

Scaling and Monitoring

Metrics: rate_limiter_pass_total, rate_limiter_reject_total, rate_limiter_retry_after_ms, rate_limiter_degraded, redis_slow_call_total, coupon_stock_remaining, coupon_issue_success_total, etc.

Prometheus alerts for Redis slow calls, local pass spikes, and HTTP latency spikes.

Fault‑Injection Drills

Four recommended drills before launch: Redis latency increase, Redis hotspot CPU saturation, core service half‑scale, Kafka backlog. Verify automatic degradation, user feedback, stock safety, and post‑activity reconciliation.

Kubernetes Deployment Tips

HPA must consider request latency, Redis slow‑call count, thread‑pool queue length, and Kafka backlog, not just CPU.

Pre‑scale to a safe water‑mark before activity, then adjust dynamically.

Set proper readiness probes, fixed JVM heap, and separate connection‑pool timeouts for Redis, MQ, DB.

User‑Facing Experience

Different failure semantics should be exposed: HTTP 429 for API callers, "Activity busy, try later" for end users, "Queued, retry in 2 s" for qualified users, and "Already claimed" for idempotent re‑tries.

Evolution Roadmap

Phase 1: Single‑node limit + DB stock (suitable for tiny activities).

Phase 2: Redis global limit + Redis pre‑deduction (mid‑scale).

Phase 3: Redis global limit + local fallback + gateway coarse limit (core production pattern).

Phase 4: Qualification pre‑warm, token queue, async reconciliation (large‑scale, multi‑channel).

Pre‑Launch Checklist

Redis slow‑path automatically switches to local fallback?

Local fallback capacity scales with pod count?

Stock has a final strong cap preventing over‑issue?

Idempotent key covers activity, template, user, request ID?

Hot coupons are sharded to avoid single‑slot hotspot?

Different responses for limit reject, out‑of‑stock, already‑claimed?

Parameters support hot update?

Fault drills for Redis slowdown, Kafka backlog, core service scaling performed?

Post‑activity stock vs. issuing reconciliation in place?

Conclusion

The essence of a production‑grade coupon‑grab system is not raw speed but bounded speed: protect entry points, allocate a budget for valuable requests, enforce a hard stock ceiling, and design deterministic degradation paths. When all four conditions hold, the system can survive millions of QPS without catastrophic failures.