When AI Rate Limiting Goes Wrong: A Four‑Dimension Framework and Three‑Layer Gateway in Practice

Midnight Incident

At 02:00 a fintech AI platform’s monitoring panel showed normal RPM, but the billing page revealed that the night’s API token consumption exceeded the total of the previous week. The root cause was a research‑analysis Agent caught in a loop: retrieve → unsatisfied result → re‑plan with LLM → retrieve again. The request rate grew from 1 req/s to 110 req/s within two minutes, while Nginx QPS throttling never triggered because each request was technically valid.

Why traditional Nginx + Redis throttling fails for AI

In classic micro‑services each request has roughly equal compute cost, so limiting by request count (QPS) is reasonable. Large‑model inference breaks this assumption. A short‑question request uses ~200 input tokens and ~50 output tokens (≈500 ms GPU time, negligible memory), whereas a long‑document analysis uses ~12 000 input tokens and ~3 000 output tokens (≈30 s GPU time, high KV‑Cache usage). Both appear as a single HTTP request to Nginx, but the GPU cluster sees dozens to hundreds of times more compute for the heavy request.

Even though RPM stayed at 80 RPM (well below a 200 RPM limit), the average of 3 500 tokens per request produced a token‑per‑minute (TPM) rate of 280 000 TPM, exceeding typical provider limits (~200 000 TPM) and causing 429 errors. This illustrates the “ghost avalanche” where RPM looks fine but TPM explodes.

Four essential dimensions of AI rate limiting

RPM (Requests Per Minute) : still useful as a safety net against client bugs or runaway Agents, but not sufficient alone.

ITPM (Input Tokens Per Minute) : measures the Prefill stage’s compute load; longer prompts increase Prefill cost and can block other requests. Anthropic separates input and output token billing for this reason.

OTPM (Output Tokens Per Minute) : governs the Decode stage’s memory‑bandwidth load; each generated token reads model weights from GPU memory, and KV‑Cache usage grows linearly. OTPM is the main cause of P99 latency spikes and is hard to predict because output length is unknown until request completion.

Cost Budget : unique to AI services that call external LLM APIs; each token has a monetary cost. In Agent scenarios a single loop can spend $40 even if the request‑count limit would have stopped it after 200 calls.

These four dimensions act at different points in the inference pipeline: RPM and ITPM intercept before the engine, OTPM monitors during generation, and Cost Budget covers the entire lifecycle.

Agent‑specific limiting controls

Agents break the assumption that traffic originates from human users with natural pacing. Problematic patterns include:

Tool‑call loops : Agent repeatedly calls a tool because the task is deemed unfinished.

Self‑propagation : An orchestrator spawns many sub‑Agents, each making further calls.

Retry storms : Downstream tool errors trigger Agent retries, which hit gateway limits, produce 429 responses, and cause further retries.

To mitigate these, three additional controls are added:

Session Token Budget : a fixed total token quota (input + output) for each Agent session; exhaustion forces termination.

Tool Call Limit : hard caps per tool type per session (e.g., max 5 searches, 10 planning calls, 3 DB writes); exceeding returns a “resource exhausted” status instead of 429.

Retry Budget : after three identical retries without progress, the session is considered stuck and stopped.

The state diagram (see image) shows the four key nodes—run, tool call, budget check, retry decision—and their termination conditions.

Three‑layer AI gateway architecture

Placing throttling logic inside each Agent leads to duplicated effort and gaps. The correct place is the gateway, the single egress point for all AI calls, organized into three layers:

Provider Compliance Layer : enforces external LLM provider quotas (RPM/ITPM/OTPM) using a Redis sliding‑window counter and atomic Lua script to deduct tokens before each request.

# Redis + Lua atomic token consumption check (example)
DEDUCT_TOKENS_SCRIPT = """
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local cost = tonumber(ARGV[2])
local window = tonumber(ARGV[3])
local current = tonumber(redis.call('GET', key) or 0)
if current + cost > limit then
    return -1  -- limit exceeded
end
redis.call('INCRBY', key, cost)
redis.call('EXPIRE', key, window)
return current + cost
"""

Tenant Isolation Layer : allocates independent token pools per team, user, or application, preventing a single large job from exhausting all tokens for other tenants. Suggested design: set limits on three dimensions (user, team, app) and enforce the strictest.

AI Safety Layer : detects AI‑specific anomalies such as sudden token‑rate spikes, repeated similar prompts, or excessive output tokens, and triggers alerts with graceful degradation instead of raw 429 responses.

The diagram (see image) illustrates the non‑overlapping responsibilities of the three layers.

Semantic cache: a helpful but tricky companion

Deploying a semantic cache can cut actual inference traffic by 20‑40 % in high‑repeat scenarios, boosting effective throughput by 30‑65 % without extra hardware. It works by vectorising prompts and returning cached responses when cosine similarity exceeds 0.92‑0.95.

The cache adds 10‑30 ms of vector‑search latency, which hurts real‑time dialogue where TTFT is critical. Enable semantic cache for structured, repeatable queries (FAQ, report generation) and disable it for creative or open‑ended conversations.

Multi‑turn conversations: the hidden token bomb

In a dialogue where each turn inputs 200 tokens and outputs 500 tokens, the prompt length grows linearly. By the 10th turn the input token count reaches roughly (200 + 500) × 10 = 7 000 tokens—35 × the first turn. RPM limits miss this growth; ITPM limits capture it, but many teams set ITPM thresholds based on early‑turn averages, causing unexpected throttling later.

Mitigation: after a certain token‑accumulation depth, summarise history into a short abstract instead of concatenating raw messages, reducing KV‑Cache pressure and P99 latency.

Rate‑limiting tool selection

LiteLLM – open‑source self‑hosted; supports 100+ providers, virtual key budgeting, OpenAI‑compatible API; no built‑in semantic cache; observability requires Prometheus/Grafana.

Portkey – managed SaaS; provides semantic cache (20‑40 % hit), built‑in guardrails, good developer experience; deep features require cloud service.

Bifrost – open‑source Go implementation; P99 latency extremely low (11 µs gateway overhead), stable 5 000 RPS; ecosystem early, plugin system less mature.

Kong AI Gateway – enterprise; rich plugin ecosystem, SSO/RBAC, suitable for strict compliance; heavy deployment, steep learning curve.

Selection guidelines:

Maximum provider flexibility & self‑hosted → LiteLLM.

Small‑to‑mid‑size teams needing quick production and semantic cache → Portkey.

Extreme performance, Go stack friendly → Bifrost.

Finance/healthcare with strict compliance → Kong AI Gateway or AWS Bedrock Gateway.

Three engineering judgments to avoid common pitfalls

Identify the real bottleneck first : external API calls are limited by TPM; self‑hosted clusters are limited by KV‑Cache memory and Decode throughput. Benchmark each scenario separately, using realistic mixed‑length traffic.

Alert before throttling : trigger alerts at 80 % of token‑rate thresholds and when tool‑call counts exceed three‑times the historical average, giving operators a window before 429 errors.

Include internal Agent calls in throttling : internal Agent‑to‑self‑hosted inference paths must also be rate‑limited; internal traffic is not automatically safe.

Failure boundaries the scheme cannot cover

Unpredictable output tokens : OTPM can only be measured after completion; optimistic pre‑deduction plus post‑adjustment introduces accounting error.

Prompt‑injection token explosions : crafted inputs can force models to generate extremely long outputs, exhausting OTPM; requires guardrails and content monitoring.

Streaming OTPM monitoring delay : tokens are sent incrementally, making real‑time OTPM accounting hard; some teams accept asynchronous post‑stream updates.

Conclusion

In the past decade system pressure was measured with QPS, but in the AI era the scarce resources are tokens, KV‑Cache, GPU memory, and inference budget. The midnight incident shows that runaway Agents, not traffic spikes, will dominate future failures. Designing a comprehensive AI rate‑limiting system—four‑dimensional token controls, three‑layer gateway, and agent‑aware safeguards—turns the problem from a simple throttler into a full‑blown AI infrastructure governance challenge.