Building Enterprise Gateways with OpenResty: Rate Limiting, Auth & Observability

From Three Nginx Layers to One Programmable Gateway

A mid-sized company's ingress evolved into: outer SLB with WAF, first Nginx for TLS offload, second for canary routing, third for static caching. Each new service added more location and upstream blocks plus custom auth checks. At a 400k QPS marketing spike, layer-two connections saturated, layer-three worker_connections exhausted, SLB health checks timed out — no layer knew whether a request should be admitted; they just forwarded.

The fix: collapse ingress into one programmable gateway using OpenResty (Nginx + LuaJIT) solving four problems:

Unified entry : single server handles TLS, routing, auth, rate limiting for all domains.

Rate limiting : per IP/user/API with "delay then reject" semantics, not blunt blocking.

Risk control : detect scraping, credential stuffing, crawlers; dynamic ban without reload.

Observability : per-request trace ID, latency breakdown, upstream status for pinpoint debugging.

Version Baseline & Applicability (2026-11)

Component Baselines

OpenResty : 1.29.2.x — 1.29.2.5 (Aug 2026) backports nginx ngx_http_rewrite_module buffer overflow patch (CVE-2026-9256); production must not be lower.

nginx : 1.29.x — New http2 on; directive; avoid listen ... http2.

LuaJIT : 2.1 (OpenResty branch) — Includes FFI & GC64 extensions; no separate install.

lua-resty-* libs : Bundled versions — Use OpenResty-bundled lua-resty-limit-traffic, lua-resty-redis, lua-resty-core; avoid mixing luarocks versions.

OS : Rocky/RHEL 9, Ubuntu 22.04/24.04 — Official repos provided.

Redis : 7.x — Only for distributed rate limiting.

Critical version coupling : lua-resty-core and lua-nginx-module are tightly bound. Many attempt to index a nil value (global 'ngx') errors come from luarocks overwriting bundled libs. Verify:

openresty -V 2>&1 | tr ' ' '
' | grep -E "lua-resty-core|lua-nginx-module|lua-resty-lrucache"

All paths must be under /usr/local/openresty/lualib or /usr/local/openresty/luajit; /usr/local/share/lua/5.1/ indicates luarocks contamination — clean it.

When to Use

Fine-grained rate limiting by user ID, tenant, API key (native limit_req only supports variable keys, no "delay-then-allow").

Dynamic policy changes: ban lists, thresholds, canary rules — effective instantly without reload.

Gateway-layer request rewrite, protocol conversion, aggregation with millisecond latency budget.

Unified auth & audit at ingress to avoid per-service duplication.

When NOT to Use

Complex business orchestration, long transactions, many external calls — gateway should be "light logic, fast fail".

Team lacks Lua expertise; OpenResty pitfalls lie in memory & coroutine models, hard to self-debug.

Simple reverse proxy & static serving — vanilla Nginx is simpler.

Core: How OpenResty Runs a Request

11 Request Phases

Nginx request processing is a fixed pipeline; Lua hooks into specific phases:

Client request
  ↓
[1] init_by_lua          master startup: load global config, precompile regex
  ↓
[2] init_worker_by_lua   per-worker startup: timers, health checks
  ↓
[3] ssl_certificate_by_lua TLS handshake: pick cert by SNI
  ↓
[4] set_by_lua           variable assignment (rare, perf so-so)
  ↓
[5] rewrite_by_lua       URI rewrite, redirect, internal redirect
  ↓
[6] access_by_lua        auth, rate limit, risk control (most used hook)
  ↓
[7] content_by_lua       generate response (replaces proxy_pass)
  ↓
[8] header_filter_by_lua rewrite response headers (add trace id, CORS)
  ↓
[9] body_filter_by_lua   stream rewrite response body (caution: breaks streaming)
  ↓
[10] log_by_lua          logging, metrics (must not block)
  ↓
[11] balancer_by_lua     custom upstream selection

Key constraints per phase: init_by_lua runs in master — cannot access ngx.var, cannot make network calls (event loop not started). Good for config loading, regex precompile, shared dict init. init_worker_by_lua runs per worker — can create ngx.timer.every timers for local cache refresh, periodic reporting. access_by_lua is the main hook — supports ngx.sleep, subrequests, Redis calls (all yield in coroutine, non-blocking). content_by_lua followed by header_filter / body_filter for response. header_filter cannot issue subrequests. log_by_lua cannot use ngx.sleep, ngx.location.capture (yielding APIs). Must offload via ngx.timer.at(0, fn).

Common mistake: calling Redis INCR in log_by_lua for access stats → API disabled in the context of log_by_lua*. Correct: accumulate in shared dict, worker timer flushes to Redis.

Coroutines & "Non-blocking" Truth

High concurrency comes from LuaJIT speed + coroutine model. When Lua calls ngx.sleep, ngx.location.capture, resty.redis:get, the current coroutine yields; worker processes other connections; resumes when I/O ready. "Non-blocking" means these APIs use event-driven internals , not that your code never blocks.

Operations that block the whole worker — avoid: os.execute, io.popen, io.open on large files. luasocket synchronous calls ( socket.connect doesn't use cosocket).

Hundreds-of-thousands full-table scans in request path.

Regex catastrophic backtracking ( ngx.re offers JIT & DFA modes; DFA avoids backtracking).

Diagnosis: if QPS stalls, config clean, CPU low — verify openresty -V shows --with-luajit, confirm lua_code_cache on (default on). Production must be on; dev off for live reload but off recompiles per request — 10-100x slower. Seen P99 spike to 800ms because someone baked lua_code_cache off from test into prod image.

Shared Memory Dict: Cross-worker State

lua_shared_dict

is the key infrastructure — all workers share one memory region for rate-limit counters, cache, dynamic config.

lua_shared_dict limit_req  200m;
lua_shared_dict limit_conn 20m;
lua_shared_dict cfg_cache  20m;
lua_shared_dict ban_list   50m;

Five properties to understand:

Slab allocation fragments : mixed value sizes (100B vs 10KB) cause fragmentation; actual capacity far below nominal. Split by purpose, keep value sizes uniform per dict. Monitor: free_space / capacity < 0.15 → expand or fix missing TTL leaks.

No atomic read-modify-write : incr(key,1) is atomic; get; set(v+1) is not. Concurrent loss shows as "configured 100, actual passes 130".

Expiration is lazy + active : expire sets TTL but memory reclaimed only by worker cleanup timer or eviction on new writes. Short-term high-cardinality keys (crawler rotating IPs) can OOM the dict.

No iteration : no keys() / items(). To scan all keys, maintain your own index. Dynamic blocklists need separate structure.

Single value length limit : default ~65KB; set fails with nil, "no memory". Watch when storing large JSON configs.

Cosocket: Why It Doesn't Block

ngx.socket.tcp()

, resty.redis, resty.mysql, resty.http use cosocket — non-blocking reads/writes. Three hard limits:

Connection pool lifetime bound to request : in access_by_lua, after redis:connect, if request ends without set_keepalive, connection closes; next request rebuilds. Standard pattern: acquire → operate → set_keepalive, with retry for stale pooled connections.

Cannot use in init_by_lua or log_by_lua — no cosocket support in those phases.

Pool size per worker : total connections ≈ worker_processes × pool_size. Factor into Redis maxclients planning.

Stable Redis wrapper (pseudo-code):

local redis = require "resty.redis"
local _M = { pool_size = 100 }
function _M.get(conf)
  local red = redis:new()
  red:set_timeouts(500, 500, 500)  -- connect/send/read 500ms
  local ok, err = red:connect(conf.host, conf.port)
  if not ok then return nil, "connect: " .. tostring(err) end
  if conf.password and conf.password ~= "" then
    local ok2, err2 = red:auth(conf.password)
    if not ok2 then return nil, "auth: " .. tostring(err2) end
  end
  if conf.db and conf.db > 0 then
    local ok3, err3 = red:select(conf.db)
    if not ok3 then return nil, "select: " .. tostring(err3) end
  end
  return red
end
function _M.put(red)
  local ok, err = red:set_keepalive(10000, _M.pool_size)
  if not ok then red:close() end
end
return _M

set_timeouts

three values: connect timeout short (500ms — if Redis unhealthy, fail fast), read timeout short — rate-limit failure should "allow" not "hang". Decision logic in troubleshooting chapter.

Never hardcode passwords . Use env vars via systemd drop-in:

# /etc/systemd/system/openresty.service.d/override.conf
[Service]
Environment=REDIS_PASSWORD_FILE=/etc/openresty/secrets/redis.env

Lua reads via os.getenv("REDIS_PASSWORD") in init_by_lua (once, cached in module var). File mode 600, owner nginx/root.

Deployment: From Package to Operable Cluster

Install

Prefer official APT/YUM repos; don't replace with distro nginx package (missing Lua modules, different paths).

# Rocky 9
dnf install -y dnf-plugins-core
dnf config-manager --add-repo https://openresty.org/package/rocky/openresty.repo
dnf install -y openresty openresty-resty

# Ubuntu 24.04
wget -O - https://openresty.org/package/pubkey.gpg | sudo gpg --dearmor -o /usr/share/keyrings/openresty.gpg
echo "deb [signed-by=/usr/share/keyrings/openresty.gpg] http://openresty.org/package/ubuntu noble main" |
sudo tee /etc/apt/sources.list.d/openresty.list
sudo apt update && sudo apt install -y openresty

Verify: /usr/local/openresty/bin/openresty -V Expect:

nginx version: openresty/1.29.2.5
built with OpenSSL 3.x
configure arguments: --prefix=/usr/local/openresty/nginx ... --with-http_v2_module --with-http_ssl_module ...

Missing --with-http_v2_module means minimal build — no HTTP/2.

Directory Layout

Don't stuff everything into one nginx.conf. Recommended:

/usr/local/openresty/nginx/conf/
├── nginx.conf              # skeleton, only includes
├── conf.d/
│   ├── 00-main.conf        # log format, shared dicts, map
│   ├── 10-upstreams.conf   # upstream definitions
│   ├── 20-locations.conf   # routing & business
│   └── 90-admin.conf       # internal admin (127.0.0.1 only)
└── lua/
    ├── access/
    │   ├── auth.lua
    │   ├── limit.lua
    │   └── waf.lua
    ├── lib/
    │   ├── redis.lua
    │   ├── config.lua
    │   └── log.lua
    └── init/
        ├── master.lua
        └── worker.lua

nginx.conf

skeleton:

user nginx;
worker_processes auto;
worker_rlimit_nofile 100000;

error_log /var/log/nginx/error.log warn;
pid /run/openresty.pid;

events {
  worker_connections 65535;
  multi_accept on;
  use epoll;
}

http {
  include mime.types;
  default_type application/octet-stream;

  lua_package_path "/usr/local/openresty/nginx/conf/lua/?.lua;;";
  lua_package_cpath ";;";

  lua_shared_dict limit_req  200m;
  lua_shared_dict limit_conn 20m;
  lua_shared_dict cfg_cache  20m;
  lua_shared_dict ban_list   50m;

  init_by_lua_block { require("init.master").run() }
  init_worker_by_lua_block { require("init.worker").run() }

  include conf.d/*.conf;
}

Key settings rationale: worker_rlimit_nofile 100000: gateway holds many long connections; default 1024 insufficient. Sync with sys fs.file-max and systemd LimitNOFILE. worker_connections 65535: per-worker connection cap. Theoretical concurrency ≈ worker_processes × worker_connections, limited by FDs & memory (few hundred bytes per connection + Lua coroutine stack). multi_accept on: accept multiple connections per epoll wakeup, reduces syscalls under high concurrency. lua_package_path trailing ;; keeps default search path; otherwise resty.* libs not found.

OS Tuning

Bottleneck often in kernel, not Nginx. Checklist:

ulimit -n
sysctl fs.file-max
sysctl net.core.somaxconn
sysctl net.ipv4.tcp_max_syn_backlog

If somaxconn = 128 (default), burst traffic overflows SYN queue → sporadic connection timeouts. Production values:

# /etc/sysctl.d/99-gateway.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 10000 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

tcp_tw_reuse

only matters when gateway acts as client (proxying upstream). ip_local_port_range must be large — otherwise upstream connections exhaust local ports ( Cannot assign requested address). With 60k ports, 60s TIME_WAIT, max new connections ≈ 1000/s; beyond that add machines or lower tcp_fin_timeout.

Reload Mechanics

openresty -s reload

flow:

Master receives HUP, re-reads config, inherits old listen sockets .

Master spawns new workers.

New workers accept new connections.

Old workers get graceful shutdown signal, stop accepting, wait for in-flight requests (bounded by worker_shutdown_timeout).

Old workers exit, master closes old sockets.

Key points:

Long connections delay reload : keepalive clients keep old workers alive until worker_shutdown_timeout (set 30-60s; trade-off: few long requests get cut).

Shared dict persists across reload , but init_by_lua re-runs before new master forks — don't put non-idempotent side effects there. init_worker_by_lua re-runs in new workers — timers rebuilt. If timer has "run once at startup", reload triggers extra executions.

Verify reload:

/usr/local/openresty/bin/openresty -t
/usr/local/openresty/bin/openresty -s reload
sleep 1
ps -eo pid,ppid,lstart,cmd | grep "nginx: worker" | grep -v grep

If old+new workers coexist beyond worker_shutdown_timeout, long connections stuck. Inspect:

ss -tnp | grep nginx | head -20

-t

passes but reload can still fail: cert file path errors ( -t doesn't validate file existence), pid file permissions. On failure master keeps old config; error.log shows [emerg] — must read logs, not just exit code.

Business Modules: Must-get-right Components

Unified Routing & Upstream

Gateway core: one server handles all domains, routes by Host+Path. Avoid per-business server blocks in conf.d — 200 services = 200 blocks, reload slows linearly.

Use map to map Host → upstream name, then dynamic proxy_pass.

# conf.d/10-upstreams.conf
upstream api_user_v1 {
  server 10.10.1.11:8080 max_fails=3 fail_timeout=5s;
  server 10.10.1.12:8080 max_fails=3 fail_timeout=5s;
  keepalive 128;
}
upstream api_order_v1 {
  server 10.10.2.11:8080 max_fails=3 fail_timeout=5s;
  server 10.10.2.12:8080 max_fails=3 fail_timeout=5s;
  keepalive 128;
}
upstream gateway_reject {
  server 127.0.0.1:1;  # explicit dead port for unmatched host fast-fail
}

# conf.d/20-locations.conf
map $host $upstream_by_host {
  default              gateway_reject;
  "api.example.com"    api_user_v1;
  "order.example.com"  api_order_v1;
}

server {
  listen 443 ssl;
  http2 on;
  server_name api.example.com order.example.com;

  ssl_certificate     /etc/openresty/certs/wildcard.crt;
  ssl_certificate_key /etc/openresty/certs/wildcard.key;
  ssl_protocols       TLSv1.2 TLSv1.3;
  ssl_session_cache   shared:SSL:20m;
  ssl_session_timeout 10m;

  set $limit_result "-";

  location / {
    access_by_lua_block {
      require("access.limit").run()
      require("access.auth").run()
    }

    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Request-ID $request_id;

    proxy_connect_timeout 2s;
    proxy_send_timeout 10s;
    proxy_read_timeout 10s;
    proxy_next_upstream error timeout http_502 http_503;

    proxy_pass http://$upstream_by_host;
  }
}

Gotchas:

proxy_set_header Connection "" + upstream keepalive : without clearing Connection, Nginx sends Connection: close to upstream, pool disabled, new connection per request → upstream port exhaustion, TIME_WAIT spike.

Variable proxy_pass resolves at runtime : if value not a defined upstream name, resolution fails.

map

default

must point to an existing upstream; gateway_reject to dead port gives fast 502 instead of hang.

Variable proxy_pass vs health checks : matching upstream name still uses max_fails / fail_timeout. Literal IP:Port bypasses upstream block — no retries, no pool. Don't mix modes.

Don't add non_idempotent to proxy_next_upstream : default retries only idempotent methods. POST retry may duplicate orders; app-layer should use idempotency keys.

Trace ID & Latency Breakdown

Gateway must answer "where is this request slow?". Minimal: use Nginx built-in $request_id, log phase timings in access_log.

# conf.d/00-main.conf
log_format gateway_json escape=json
'{"ts":"$time_iso8601","rid":"$request_id","host":"$host","uri":"$uri",'
'"method":"$request_method","status":$status,"bytes":$body_bytes_sent,'
'"rt":$request_time,"urt":"$upstream_response_time",'
'"uct":"$upstream_connect_time","uaddr":"$upstream_addr",'
'"ua":"$http_user_agent","ip":"$remote_addr","limit":"$limit_result"}';

access_log /var/log/nginx/access.log gateway_json buffer=64k flush=5s;

$request_time

- $upstream_response_time = gateway self-cost. Diagnosis:

Diff > 20ms consistently: gateway has blocking op or heavy Lua. $upstream_connect_time high: upstream connect slow (backlog full or network). $upstream_response_time shows comma-separated values: retries happened → upstream unstable.

Real debug: once saw rt=900ms, urt=30ms, diff 870ms in gateway. Root cause: lua_code_cache off (see §3.2). Turning on dropped P99 from 900ms to 8ms.

Use $request_id not custom generator — 16-byte hex, near-zero cost, per-machine unique. For cross-host tracing, propagate via response header:

add_header X-Request-ID $request_id always;

always

ensures error responses (4xx/5xx) also carry the ID — those are exactly the ones you need most.

Dynamic Config: Change Thresholds Without Reload

Gateway policy params (rate limits, ban switches, canary ratios) must be mutable. Use shared dict + periodic refresh.

-- lua/lib/config.lua
local _M = {}
local shm = ngx.shared.cfg_cache
local DEFAULT = {
  limit_rps   = 100,
  limit_burst = 50,
  waf_enabled = true,
  gray_percent = 0,
}

local function load_from_remote()
  local res = ngx.location.capture("/_internal/config", { method = ngx.HTTP_GET })
  if res.status ~= 200 then
    ngx.log(ngx.ERR, "fetch config failed, status=", res.status)
    return false
  end
  local cjson = require "cjson.safe"
  local obj = cjson.decode(res.body)
  if not obj then
    ngx.log(ngx.ERR, "config json invalid")
    return false
  end
  for k, v in pairs(obj) do shm:set(k, v) end
  shm:set("_loaded_at", ngx.now())
  return true
end

function _M.get(key)
  local v = shm:get(key)
  if v == nil then return DEFAULT[key] end
  return v
end

function _M.start_refresh(interval)
  local ok, err = ngx.timer.every(interval, function(premature)
    if premature then return end
    local ok2, err2 = pcall(load_from_remote)
    if not ok2 then ngx.log(ngx.ERR, "refresh config error: ", err2) end
  end)
  if not ok then ngx.log(ngx.ERR, "start config timer failed: ", err) end
end

return _M

Key points:

No network in init_by_lua — event loop not up; ngx.location.capture fails. Use init_worker_by_lua to create timer.

Timer callback must pcall — uncaught error kills the timer permanently.

Defaults required — shared dict empty after reload until timer runs first round; can't return nil.

ngx.location.capture only hits local location — /_internal/config is a proxy_pass to config center on same host.

Verify:

curl -s http://127.0.0.1:8080/_admin/config | jq .

Admin endpoint must listen only on 127.0.0.1 with access control — otherwise attack surface.

Auth Skeleton

Order matters: authenticate (who) → rate limit (how much) → authorize (can do). Reverse order leaks quota to unauthenticated or bypasses auth.

-- lua/access/auth.lua
local cjson = require "cjson.safe"
local cfg = require "lib.config"

local _M = {}

local function extract_key(headers)
  local auth = headers["authorization"]
  if auth then
    local token = auth:match("^[Bb]earer%s+(.+)$")
    if token then return token end
  end
  return headers["x-api-key"]
end

local function verify_api_key(key)
  local shm = ngx.shared.cfg_cache
  local info = shm:get("apikey:" .. key)
  if not info then return nil, "unknown key" end
  return cjson.decode(info)
end

function _M.run()
  local path = ngx.var.uri
  if path == "/healthz" then return end

  local raw = extract_key(ngx.req.get_headers())
  if not raw then
    ngx.status = 401
    ngx.header["Content-Type"] = "application/json"
    ngx.say('{"code":401,"msg":"missing credential"}')
    return ngx.exit(401)
  end

  local key = verify_api_key(raw)
  if not key then
    ngx.status = 401
    ngx.header["Content-Type"] = "application/json"
    ngx.say('{"code":401,"msg":"invalid credential"}')
    return ngx.exit(401)
  end

  ngx.ctx.consumer   = key.consumer
  ngx.ctx.tier       = key.tier
  ngx.ctx.quota_rps  = key.rps or cfg.get("limit_rps")
end

return _M

Uses ngx.ctx not ngx.var for inter-module data — ngx.var hits Nginx variable system (stringify, hash lookup), measurable overhead at 10k+ QPS. ngx.ctx is plain Lua table.

Pitfall : subrequests have independent ctx; ngx.location.capture modifications don't affect parent. If auth runs in subrequest, explicitly pass results back.

API Key storage : above reads from shared dict; production dict populated by control plane via admin API. Never store plaintext keys in Lua files or config — they'd enter Git. Store sha256(key) prefix (16 bytes) + permissions; dict leak won't yield usable keys.

Rate Limiting: Three Algorithms, Four Implementations

Clarify What You're Limiting

Rate (QPS) : Anti-scrape, anti-crawl — key: IP / User ID / API Key — storage: Shared dict or Redis — on exceed: Delay or reject.

Concurrency (connections) : Prevent slow requests exhausting pool — key: IP / Upstream — storage: Shared dict — on exceed: Reject.

Quota (daily/monthly total) : Plan metering — key: Tenant ID — storage: Redis (persistent) — on exceed: Reject with remaining.

Bandwidth : Prevent large files saturating egress — key: IP — storage: Native limit_rate — on exceed: Throttle.

Four distinct algorithms — don't mix in one module.

Leaky Bucket: Smooth Output, Adds Latency

resty.limit.req

implements leaky bucket. new(shdict_name, rate, burst):

Request rate ≤ rate: immediate pass, delay=0, 2nd return 0.

Rate between rate and rate+burst: returns delay (seconds), 2nd return = excess rate. Caller ngx.sleep(delay) then passes; output smoothed to rate.

Exceeds rate+burst: returns nil, "rejected".

-- lua/access/limit.lua (leaky bucket part)
local limit_req = require "resty.limit.req"
local cfg = require "lib.config"

local _M = {}
local lim = limit_req.new("limit_req", cfg.get("limit_rps"), cfg.get("limit_burst"))

local function limit_by_rate(key, name)
  local delay, err = lim:incoming(key, true)
  if not delay then
    if err == "rejected" then
      ngx.header["Retry-After"] = "1"
      ngx.var.limit_result = name .. ":reject"
      ngx.status = 429
      ngx.header["Content-Type"] = "application/json"
      ngx.say('{"code":429,"msg":"too many requests"}')
      return ngx.exit(429)
    end
    -- internal error → allow, don't block business due to limiter bug
    ngx.log(ngx.ERR, "limit req internal error: ", tostring(err))
    ngx.var.limit_result = name .. ":error"
    return
  end
  if delay >= 0.001 then
    ngx.var.limit_result = name .. ":delay"
    ngx.sleep(delay)
  end
end

_M.limit_by_rate = limit_by_rate
return _M

Critical decision : whether to ngx.sleep(delay). Doing it makes client feel "slow" not "failed" — better UX but consumes connections. Example: rate=100, burst=50, worst-case 50 requests sleeping avg 0.5s = 25 connections occupied. For connection-sensitive gateways, count this cost.

Skipping sleep degrades to "token bucket + reject": excess → immediate 429. API callers understand 429 + backoff; sleep makes them think service slowed, triggers their own timeout/retry, amplifying traffic.

Recommendation: internal APIs → delay; public user traffic → reject . Internal callers handle 429 + backoff; public users prefer "slow" over error, and rarely retry. incoming 2nd param commit: false = dry-run (compute only, no record). Great for canary — run dry-run a day, watch limit_result distribution, confirm thresholds before enabling.

Instance must be created in init_by_lua or module top-level , not per-request. new() does dict lookup & validation; per-request wastes cycles. Above code stores lim as module variable, created on first require.

Token Bucket: Allows Burst, DIY Required

Leaky bucket's "smooth" hurts bursty legit traffic (e.g., client normally 1 QPS, occasionally submits 20 at once). Leaky bucket delays them 20s — bad UX. Token bucket permits short bursts.

OpenResty has no built-in token bucket ( resty.limit.count is fixed-window counter). DIY with shared dict storing current tokens + last refill time.

-- lua/lib/token_bucket.lua
local _M = {}

-- Use incr atomicity for "take token", avoid read-modify-write race
-- Structure: key = available tokens (int), key..":t" = last refill timestamp
function _M.acquire(shm, key, rate, capacity)
  local now = ngx.now()
  local tokens_key = "tb:" .. key
  local ts_key = tokens_key .. ":t"

  -- Step 1: refill. add only writes if key absent, ensures init once
  local ok, err = shm:add(ts_key, now, 2)
  if not ok and err ~= "exists" then return nil, err end
  local last = shm:get(ts_key) or now
  local elapsed = math.min(now - last, 1.0)  -- cap refill to 1s, prevent idle burst dump
  local refill = math.floor(elapsed * rate)
  if refill > 0 then
    local newv = shm:incr(tokens_key, refill, 0)
    if newv and newv > capacity then shm:set(tokens_key, capacity) end
    shm:set(ts_key, now)
  end

  -- Step 2: take one token (incr -1 atomic, prevents over-grant)
  local left = shm:incr(tokens_key, -1, 0)
  if not left then return nil, "incr failed" end
  if left < 0 then
    shm:incr(tokens_key, 1)  -- return borrowed token, avoid negative accumulation
    return false, math.abs(left)
  end
  return true, left
end

return _M

Design notes:

Integer tokens : incr only supports numbers; float accumulation has precision issues + non-atomic set. Store as integer (1 token = 1 unit), use incr for atomicity.

Refill capped at 1s : idle 10min → rate × 600 tokens dumped, then instant burst of capacity requests. That's not "burst", it's "backlog release". 1s cap = tolerate at most 1s burst.

incr(-1) not get +check : latter over-grants under concurrency. incr returns negative → borrowed, immediately incr(+1) returns. Tiny race (might return someone else's token) but error ≤1 token, direction "under-allow" — acceptable.

Limitation : incr with init param atomically creates missing key (3rd arg = init value). Behavior consistent across OpenResty versions.

Alternative: resty.limit.count (fixed-window counter):

local limit_count = require "resty.limit.count"
local lim = limit_count.new("limit_req", 100, 1)  -- 100/sec
local ok, err = lim:incoming(key, true)
if not ok then
  if err == "rejected" then return ngx.exit(429) end
end

Fixed-window edge problem: window 1s, limit 100. Client sends 100 at 0.99s, 100 at 1.01s → 200 in 0.02s. Fine for anti-scrape; insufficient for strict rate control. Token bucket has no boundary issue.

Distributed Rate Limiting: When Redis Is Mandatory

Single-machine shared dict only works per-instance. 10 gateways each rate=100 → cluster allows 1000. Often acceptable (protecting backend; backend total capacity is baseline, split evenly).

Only three cases require Redis:

Quota limits : user daily 1000 calls — must be globally exact.

Ban lists : block an IP everywhere instantly.

Few/uneven nodes : e.g., 3 gateways, LB unbalanced — per-node limiting starves hot node, cold node idle.

Redis sliding-window with Lua script for atomicity:

-- lua/lib/redis_limit.lua
local redis = require "lib.redis"

local SCRIPT = [[
local key   = KEYS[1]
local now   = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])

redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
local count = redis.call('ZCARD', key)
if count >= limit then return 0 end
redis.call('ZADD', key, now, ARGV[4])
redis.call('PEXPIRE', key, window)
return 1
]]

local _M = {}
function _M.allow(key, limit, window_ms)
  local red, err = redis.get({ host = "127.0.0.1", port = 6379 })
  if not red then return nil, err end
  local sha = red:script_load(SCRIPT)
  if not sha then redis.put(red); return nil, "script_load failed" end
  local now_ms = ngx.now() * 1000
  local member = string.format("%.0f-%s", now_ms, tostring(math.random(1, 1e6)))
  local res, err2 = red:evalsha(sha, 1, key, now_ms, window_ms, limit, member)
  redis.put(red)
  if not res then return nil, tostring(err2) end
  return res == 1
end
return _M

Key points:

Use ZSET not INCR + EXPIRE : INCR is fixed-window with boundary flaw; ZSET stores per-request timestamp → precise "last N ms ≤ M". Cost: memory ∝ limit — if limit=10000, each key holds 10k members; tens of keys eat GBs. Sliding window only for small limits (few hundred).

Cache script_load SHA in module var; reload only on NOSCRIPT error.

Redis failure policy : above returns nil on connect failure; caller decides allow (degrade) or reject (strict) — see troubleshooting ch.

Hot key mitigation : single attacked IP hits one Redis shard. In Cluster, consider key % 16 scatter + aggregate (loses precision). Practical: local pre-filter + Redis fallback — shared dict coarse limit (2× threshold), only excess hits Redis. 99% normal traffic never touches Redis.

Concurrency Limiting: Stop Slow Requests Eating Connections

QPS limit doesn't stop "every request is slow". 100 QPS × 30s = 3000 concurrent connections → gateway saturated. Concurrency limit separate.

-- lua/access/limit_conn.lua
local limit_conn = require "resty.limit.conn"

local _M = {}
local lim = limit_conn.new("limit_conn", 20, 5, 0.5)  -- 20 concurrent, 5 delay buffer, avg 0.5s

function _M.run(key)
  local delay, err = lim:incoming(key, true)
  if not delay then
    if err == "rejected" then
      ngx.var.limit_result = "conn:reject"
      return ngx.exit(429)
    end
    return
  end
  if delay > 0 then ngx.sleep(delay) end
  -- CRITICAL: must return on request end, else counter only increments
  lim:leaving(key)
end

return _M

resty.limit.conn

incoming

increments shared dict; leaving decrements. Missing leaving → counter only grows → all requests rejected after minutes. But calling incoming in access_by_lua and leaving at function end limits "requests in access phase", not full request concurrency.

Correct: use ngx.on_abort or log_by_lua for paired release. Simpler: let resty.limit.conn manage lifecycle — it registers callback at request end (some versions). If not, call leaving in log_by_lua:

log_by_lua_block { require("access.limit_conn").leave(ngx.var.remote_addr) }

But log_by_lua can't retrieve key if based on user ID not IP. Robust: store key in ngx.ctx at ingress, read in log_by_lua:

-- access phase
ngx.ctx.conn_key = key
lim:incoming(key, true)

-- log phase
local key = ngx.ctx.conn_key
if key then lim:leaving(key) end

This pairing is the most missed step in rate limiting. Symptom: "works at launch, hours later all 429, restart fixes" — counter only increments, never decrements, eventually saturates. If you see this, check leaving pairing first.

Multi-dimensional Combination: resty.limit.traffic

One request may need "IP 100 QPS" AND "User 10 QPS". Separate incoming calls cause leak: first passes, second rejects, but first's counter already incremented with no rollback. resty.limit.traffic combines limiters atomically — all pass → commit; any fail → uncommit all.

local limit_req = require "resty.limit.req"
local limit_traffic = require "resty.limit.traffic"

local lim_ip   = limit_req.new("limit_req", 100, 20)
local lim_user = limit_req.new("limit_req", 10, 5)
local limiters = { lim_ip, lim_user }

local delays, err = limit_traffic.combine(limiters, { ip_key, user_key }, true)
if not delays then
  if err == "rejected" then return ngx.exit(429) end
  return
end
for _, d in ipairs(delays) do
  if d >= 0.001 then ngx.sleep(d) end
end

Order of limiters must match keys. Same shared dict OK but separate dicts recommended ( limit_req_ip, limit_req_user) — different key cardinality & access frequency; mixing worsens fragmentation.

Setting Thresholds: Not Guesswork

Find backend true capacity : load-test core endpoints, locate P99 degradation point, take 70% as backend capacity. Not "max QPS" but "stable SLA QPS".

Allocate by business : backend 2000 QPS, 5 endpoints sharing. Distribute by historical traffic share, each gets 20% headroom. E.g., order endpoint 40% → 2000 × 0.4 × 1.2 = 960.

Tier by client : free vs paid tiers differ. Auth phase sets tier field:

local tier_rps = { free = 10, pro = 100, enterprise = 1000 }
local rps = tier_rps[ngx.ctx.tier] or tier_rps.free

Canary validate : dry-run ( commit=false), measure rejection rate. >0.1% → threshold too low or anomaly traffic.

Emergency lane : events can temporarily raise specific keys (via dynamic config) but require approval & auto-expiry. Seen incident: pre-sale threshold raised 10×, forgot to revert → scraped for 3hrs next week.

Response Status Codes

Affects client retry behavior.

429 Too Many Requests : standard, with Retry-After header telling client when to retry. Best for APIs.

503 Service Unavailable : means "service temporarily down" (backend overload, restart). Using for rate limiting misleads monitoring as service failure.

200 + business error code : legacy systems do this; pro: client won't retry; con: monitoring blind. Not recommended.

Recommended: 429 + Retry-After:

ngx.header["Retry-After"] = "1"
ngx.header["X-RateLimit-Limit"] = tostring(rps)
ngx.header["X-RateLimit-Remaining"] = "0"

X-RateLimit-*

not standard but de-facto; lets caller compute backoff, reduces useless retries.

For delayed (not rejected) requests, add header so caller senses it:

ngx.header["X-RateLimit-Delayed"] = string.format("%.3f", delay)

vs Native ngx_http_limit_req_module

Key : Native only variable combos; Lua arbitrary Lua expr (Redis lookup, business ID concat).

Dynamic params : Native requires reload; Lua runtime change.

Precision info : Native only $limit_req_status; Lua delay, excess values.

Multi-dim combine : Native multiple zones, independent; Lua resty.limit.traffic atomic combine.

Performance : Native highest (C); Lua high (LuaJIT), ~1-2µs/call.

Verdict: if key is just $binary_remote_addr , threshold fixed → native limit_req faster & stabler . Need business ID, dynamic, combine → Lua. Can coexist: outer native for coarse DDoS protection, inner Lua for fine control.

Risk Control: From Rate Limiting to Detection

Rate limiting = "block by rule"; Risk control = "identify by behavior". Difference: rate limiting only counts; risk control examines patterns.

Three Attack Types & Signals

CC / API flooding : few URLs spike QPS, single UA, missing Referer/Cookie, highly regular intervals (variance <5ms). Signals: per-IP interval stddev <5ms, UA global frequency below threshold, URLs concentrated on few endpoints.

Credential stuffing : POST to login, many usernames, focused passwords, success rate <1%. Signals: per-IP login failure rate >80%, high username uniqueness, uniform request body size.

Crawlers : traverse URL space, regular intervals, missing headers ( Accept-Language, Sec-Fetch-*), or forged UA with mismatched TLS fingerprint.

Local Counting + Rule Engine

Simplest: sliding counters + threshold rules. Counters in shared dict, rules from dynamic config.

-- lua/access/waf.lua
local cjson = require "cjson.safe"
local cfg = require "lib.config"

local _M = {}
local shm = ngx.shared.ban_list

-- Count & return current window hits. Fixed 60s window, minute-aligned
local function hit(key, window)
  local bucket = math.floor(ngx.time() / window)
  local k = key .. ":" .. bucket
  local n, err = shm:incr(k, 1, 0)
  if not n then return 0, err end
  if n == 1 then shm:expire(k, window * 2) end  -- expire 2 windows, prevent boundary loss
  return n
end

-- Ban: write dict with value = unban timestamp
local function ban(ip, seconds, reason)
  local until_ts = ngx.time() + seconds
  shm:set("ban:" .. ip, until_ts, seconds)
  ngx.log(ngx.WARN, "ban ip=", ip, " reason=", reason, " until=", until_ts)
end

local function is_banned(ip)
  local ts = shm:get("ban:" .. ip)
  if not ts then return false end
  if ts < ngx.time() then shm:delete("ban:" .. ip); return false end
  return true
end

function _M.run()
  if not cfg.get("waf_enabled") then return end
  local ip = ngx.var.remote_addr
  if is_banned(ip) then
    ngx.var.limit_result = "waf:banned"
    return ngx.exit(403)
  end

  -- Rule 1: same IP >50 4xx/min → ban 10min
  if ngx.status >= 400 then
    local n = hit("err4xx:" .. ip, 60)
    if n > 50 then ban(ip, 600, "too_many_4xx"); return ngx.exit(403) end
  end

  -- Rule 2: login endpoint >10 failures/min → ban 30min
  if ngx.var.uri == "/api/login" and ngx.req.get_method() == "POST" then
    local n = hit("login:" .. ip, 60)
    if n > 10 then ban(ip, 1800, "login_bruteforce"); return ngx.exit(403) end
  end
end

return _M

Critical details: shm:incr(k, 1, 0) 3rd arg = initial value if missing (0 = start from 0). Atomic, safer than get + set. shm:expire only on first write ( n==1). Setting every hit resets TTL → never expires if hit every second.

Aligned window vs rolling : above uses minute-aligned fixed window — boundary issue exists (59s + 61s each 50 hits, both under threshold). Acceptable for risk control: attacks sustain, don't sit on boundary. For precision, use Redis sliding window (§6.4).

Ban storage : local shared dict = per-node ban. Node A bans, Node B unaware. ≤5 nodes acceptable (attacker hits all, all ban quickly). More nodes → write Redis, sync to each node's dict via init_worker timer.

Tiered Response: Observe → Challenge → Throttle → Ban

Immediate ban too blunt — hurts NAT users (one corporate IP blocked = whole company down). Tiered:

Observe : Count exceeds soft threshold → Tag only, no request impact.

Challenge : Exceeds mid threshold → Return JS challenge / CAPTCHA → Bots fail, humans seamless.

Throttle : Exceeds hard threshold → Reduce rate limit → Legit users slightly slower.

Ban : Clear attack signature → 403 for period → IP fully blocked.

local level = "observe"
if n > soft_limit then level = "observe"
elseif n > mid_limit then level = "challenge"
elseif n > hard_limit then level = "throttle"
end

if level == "challenge" then
  return require("waf.challenge").run()
elseif level == "throttle" then
  return require("access.limit").limit_by_rate("slow:" .. ip, "waf")
end

JS challenge implementation : must return 200 + JS page; JS computes token, sets cookie, re-requests. Can't just ngx.exit in access_by_lua — need body output via content_by_lua or subrequest. Challenge page must be uncacheable (no CDN caching).

Compliance Boundaries

Don't collect non-security data. Browser/Canvas fingerprinting requires user consent in many jurisdictions.

Logs with IP, UA, request body are personal data — retention per policy, typically ≤6 months.

Ban must have appeal path. False-positive recovery process must exist or becomes support ticket storm.

Rules auditable: who blacklisted which CIDR when — logged (hence ban logs).

Performance: Gateway Latency Budget

Latency Breakdown

Client → [TLS handshake] → [Recv request] → [access Lua] → [Connect upstream] → [Wait upstream] → [Forward response]

Gateway-controllable: access Lua + forwarding. Reference (same DC, 1000 QPS, small body):

Pure proxy (no Lua): P50 ~0.5ms.

+1 shared dict read: +10-20µs.

+1 Redis call: +200-800µs (incl. RTT).

+1 subrequest: +1-3ms (depends on subrequest).

+1 cjson.decode 1KB JSON: +20-50µs.

Conclusion: access_by_lua can afford only ONE network call . Auth queries Redis, rate limit queries Redis, config queries Redis → 2ms+ added. Fix: merge via pipeline or Lua script — single Redis round-trip fetches all needed data.

red:init_pipeline()
red:hmget("apikey:" .. key, "consumer", "tier", "rps")
red:get("config:limit_rps")
red:get("ban:" .. ip)
local res, err = red:commit_pipeline()
-- parse res[1][1], res[1][2], res[1][3], res[2], res[3]

Pipeline saves RTTs (3→1), latency 2.4ms→0.8ms. Server execution time unchanged.

Load Testing Method & Baselines

Don't use ab — connection reuse & HTTP/1.1 pipelining differ from modern clients. Use wrk or hey.

# 4 threads, 400 connections, 30s, keepalive
wrk -t4 -c400 -d30s --latency http://127.0.0.1:8080/api/ping

Sample output:

Running 30s test @ http://127.0.0.1:8080/api/ping
  4 threads and 400 connections
  Thread Stats   Avg     Stdev     Max   +/- Stdev
    Latency   3.21ms   1.87ms  42.10ms   88.31%
    Req/Sec  31.42k    2.11k  38.40k   71.24%
  Latency Distribution
    50%   2.85ms
    75%   3.62ms
    90%   4.85ms
    99%  10.20ms
  3754210 requests in 30.00s, 512.30MB read
Requests/sec: 125140.33

How to read:

125k QPS, P99 10ms : healthy single-node gateway. P99 >50ms → blocking or GC pressure.

Req/Sec Stdev high : periodic jitter, often Lua GC or timers.

Baseline comparison : first test empty location /ping { return 200 "ok"; } for native baseline, then add Lua — delta = your code cost. Baseline typically 80-150k QPS; each dict read costs 5-10%.

Three must-dos during load test:

Client machine separate from gateway — wrk caps ~30-50k QPS single-box.

Watch error.log for too many open files, worker_connections are not enough → config insufficient.

Capture system metrics: top -H for per-worker CPU balance; mpstat -P ALL 1 for core utilization (few hot cores = wrong worker_processes).

Lua GC & Memory

LuaJIT GC incremental but has small STW pauses → P99 periodic spikes.

Reduce GC pressure:

No large tables in request path. ngx.req.get_post_args() parses whole body; if only need one field, use ngx.req.read_body() + string match.

Reuse module-level tables. Hot constants (status messages, errors) at module top, not per-request {}.

Avoid string concat chains a..b..c..d — creates intermediate strings. Use table.concat. ngx.ctx only essential fields; large data → nil after use.

Monitor GC:

local jit = require "jit"
ngx.say("jit_on=", tostring(jit.status()))
collectgarbage("collect")
ngx.say("lua_mem_kb=", math.floor(collectgarbage("count")))

Production jit.status() must be true. false = JIT disabled (compile-time missing or jit.off() called) → 5-10× slower.

Troubleshooting: Six Scenarios Closed-loop

Each follows: Phenomenon → Initial Diagnosis → Checks → Metrics → Root Cause → Fix → Verify → Rollback → Postmortem.

9.1 P99 Spikes 8ms → 900ms

Phenomenon : Alert shows gateway P99 8ms→900ms, QPS normal, error rate normal, upstream latency normal.

Initial : request_time - upstream_response_time gap large → issue in gateway, not upstream.

Checks :

# 1) Confirm gap
tail -10000 /var/log/nginx/access.log | jq -r '.rt .urt' | awk '{d=$1-$2; s+=d; if(d>m)m=d} END{print "avg_gap="s/NR, "max_gap="m}'
# avg_gap > 0.02 → gateway

# 2) Config changed?
/usr/local/openresty/bin/openresty -T 2>/dev/null | grep -E "lua_code_cache|worker_processes"
# Expect: lua_code_cache on; if off → root cause

# 3) Worker CPU balanced?
top -H -p $(pgrep -f "nginx: master" | head -1)

# 4) Blocking calls in error.log
grep -c "API disabled\|no resolver" /var/log/nginx/error.log

Metrics : lua_code_cache off → every request stat + reloads Lua files, tens-hundreds µs each, frequent GC. 8 workers doing it → latency spikes.

Root cause : Config error (test config promoted to prod).

Fix : Set lua_code_cache on, openresty -t && openresty -s reload.

Verify : 5min observe P99 <10ms; wrk spot-check.

Rollback : This is rollback to normal; no rollback needed. But record in change system, add CI lint: prod conf.d must not contain lua_code_cache off.

Postmortem : Configs should be managed by config tool, not manual. Add lint: prod conf.d forbids perf-sensitive off values.

9.2 All Requests 429, Restart Fixes (Recurs Every 2hrs)

Phenomenon : 2hrs after deploy, all requests 429; restart fixes; 2hrs later repeats.

Initial : Rate-limit counter leak — only increments, never decrements. Check leaving pairing.

Checks :

# Dict capacity & free
local shm = ngx.shared.limit_conn
ngx.say("cap=", shm:capacity(), " free=", shm:free_space(), " ratio=", shm:free_space()/shm:capacity())
# ratio steadily dropping, no recovery → keys not collected

# 429 distribution
tail -5000 /var/log/nginx/access.log | jq -r 'select(.status==429) | .limit' | sort | uniq -c
# If all "conn:reject" → concurrency limiter

Metrics : resty.limit.conn counter in shared dict. incoming increments, no leaving → linear growth → threshold hit → all rejected.

Root cause : incoming / leaving unpaired. Common bug:

lim:incoming(key, true)
do_something()  -- may ngx.exit(401)
lim:leaving(key)  -- never reached

Fix : Store key in ngx.ctx, release in log_by_lua (runs regardless of exit path):

-- access
ngx.ctx.conn_key = key
lim:incoming(key, true)

-- log
local key = ngx.ctx.conn_key
if key then lim:leaving(key) end

Verify : free_space stabilizes (stops dropping), 429 rate returns to baseline.

Rollback : If fix increases 429 (double leaving), revert & raise threshold temporarily, then debug.

Postmortem : All counter-based limiters need automated test: run 1000 requests (incl. error branches) in test env, assert shared dict counters return to zero.

9.3 Shared Dict Full, New Keys Fail

Phenomenon : no memory in logs, rate limiting ineffective (all pass), free_space near zero.

Initial : Key cardinality explosion or oversized values.

Checks :

grep -c "no memory" /var/log/nginx/error.log
local shm = ngx.shared.limit_req
ngx.say("cap=", shm:capacity(), " free=", shm:free_space())

Metrics : free_space <0.05 + no memory logs.

Root cause : (1) IP-based limiting under massive distinct IPs (crawler/attack), no TTL; (2) set large values (entire config JSON per key).

Fix :

All rate-limit keys must have TTL. resty.limit.req auto-expires; custom counters need explicit expire.

Expand dict: lua_shared_dict limit_req 200m; → larger, reload (dict rebuilt, existing counters lost — acceptable).

Pre-aggregate high-cardinality dims: /24 CIDR prefix reduces key count.

Verify : Reload → free_space >80%, no memory stops.

Rollback : Dict expansion not rolled back (would re-fill). If OOM, shrink other dicts.

Postmortem : Monitor free_space/capacity <20% → alert. Must be day-one monitoring.

9.4 Dynamic Config Changed But Not Effective

Phenomenon : Control plane changes limit_rps 100→200; 10min later, 429 still at 100-rate.

Initial : Config not refreshed into shared dict, or code reads hardcoded value.

Checks :

# 1) Read live dict value
curl -s http://127.0.0.1:8080/_admin/config | jq '.limit_rps'
# Still 100 → refresh failed

# 2) Timer running?
grep -c "refresh config error" /var/log/nginx/error.log

# 3) Config center response
curl -s http://127.0.0.1:8080/_internal/config | head -c 200

Metrics : cfg_cache:_loaded_at timestamp should update every N seconds. Stale → timer dead.

Root causes (by probability):

Timer callback threw, ngx.timer.every stopped : load_from_remote lacks pcall; one cjson.decode failure kills timer permanently. Most common.

limit_req.new() captured rate at module load : resty.limit.req instance takes rate / burst at creation; later must call lim:set_rate() / set_burst(). Changing shared dict alone doesn't affect existing instance.

Per-worker module vars not shared : Each worker has independent lim; if only one worker calls set_rate, others stay stale.

Fix : Sync limiter params from dynamic config before each limit check (or periodic refresh):

local rps = cfg.get("limit_rps")
if lim.rate ~= rps then
  lim:set_rate(rps)
  lim:set_burst(math.max(1, math.floor(rps * 0.5)))
end
local delay, err = lim:incoming(key, true)

Note: lim.rate internal field name may vary; safer to cache last-set value in module:

local last_rps = -1
local function sync_rate(rps)
  if last_rps ~= rps then
    lim:set_rate(rps)
    lim:set_burst(math.max(1, math.floor(rps * 0.5)))
    last_rps = rps
  end
end

Verify : Change config, wait 30s (refresh interval), wrk burst — 429 frequency should shift.

Rollback : Config change reversible anytime. But if bad config causes mass 429, adjust via admin API — don't reload (reload resets shared dict counters → instant traffic surge).

Postmortem : All ngx.timer callbacks MUST pcall — hard rule. Document dynamic config effective latency (refresh interval + limiter sync delay) to avoid ops misjudgment.

9.5 Mass 502 But Upstream Healthy

Phenomenon : Gateway returns many 502; upstream monitoring shows healthy, low CPU, normal QPS.

Initial : Not upstream service — gateway-to-upstream connection layer.

Checks :

# 1) Upstream fields
tail -1000 /var/log/nginx/access.log | jq -r 'select(.status==502) | "\(.uaddr) \(.urt)"' | sort | uniq -c | sort -rn | head
# uaddr empty → connection never established (upstream_addr empty or -) = connect-phase fail

# 2) error.log specifics
grep "connect() failed\|no live upstreams\|upstream prematurely closed" /var/log/nginx/error.log | tail -20

# 3) Local ports & connections
ss -s
ss -tan state time-wait | wc -l

Metrics : upstream_addr empty + connect() failed (111: Connection refused): upstream not listening or wrong port. connect() failed (99: Cannot assign requested address): local port exhaustion. no live upstreams: all upstreams marked down by max_fails. upstream prematurely closed connection: upstream accepted then closed — upstream connection limit hit or process restarting.

Root causes & fixes :

Port exhaustion : ip_local_port_range too small or too many TIME_WAIT. ss -s check timewait count; >30k → act.

no live upstreams : fail_timeout too long (e.g., 60s) — one blip marks all nodes down, subsequent requests rejected until timeout. Set fail_timeout=5-10s, max_fails=2-3.

prematurely closed : upstream backlog too small, or upstream keepalive timeout mismatched. Fix upstream config:

upstream api_user_v1 {
  server 10.10.1.11:8080 max_fails=2 fail_timeout=5s;
  server 10.10.1.12:8080 max_fails=2 fail_timeout=5s;
  keepalive 128;
  keepalive_timeout 60s;  # shorter than upstream idle timeout, avoid using connection upstream already closed
  keepalive_requests 1000;
}

keepalive_timeout

must be < upstream idle close time; otherwise gateway gets already-closed connection → first request fails → retry. These 502s are "sporadic, low rate, retry succeeds" — easily dismissed as network jitter.

Verify : 502 rate → 0, connection errors disappear from error.log.

Rollback : Upstream params adjustable anytime, reload (low-peak; reload clears connection pool → burst of new connections).

Postmortem : All upstreams should explicitly set keepalive_timeout; don't rely on defaults. Alert on 502 rate >0.1% for 1min.

9.6 Reload Succeeds But Config Not Effective

Phenomenon : openresty -t passes, openresty -s reload returns 0, behavior unchanged.

Initial : Reload actually failed (old config still running), or effective config differs from expectation.

Checks :

# 1) Latest error.log
tail -30 /var/log/nginx/error.log

# 2) Worker start times
ps -eo pid,lstart,cmd | grep "nginx: worker" | grep -v grep
# If start time old → reload didn't take

# 3) Dump actual merged config
/usr/local/openresty/bin/openresty -T 2>/dev/null | grep -A5 "location /" | head -40
# -T outputs FULL merged config — only reliable way to verify "what's actually live"

Metrics : Worker lstart,

error.log

signal process started

/ reconfiguring.

Root causes :

-t passes but reload fails : cert file missing, pid file unwritable, log dir permission. -t only syntax + partial resource check; misses some runtime deps.

Edited wrong file : Two files in conf.d define same upstream, or include order causes later to override earlier.

Config outside http block : e.g., map inside server — Nginx ignores or errors.

worker_processes mismatch : Changed 4→8 but system nproc limit 4 → actual stays 4.

Fix : Address each [emerg] in error.log. For permissions, check /var/log/nginx & /run ownership.

Verify : ps confirms all workers new; openresty -T confirms config; curl -I probe request matches new config.

Rollback : Reload failure = old config still running, no rollback needed. If new config works but buggy, revert file & reload. Note: rapid reloads accumulate old workers (if long connections persist); limit with worker_shutdown_timeout.

Postmortem : Post-reload auto-check mandatory: script diffs -T output vs expected, checks worker start time, fires probe request. Three steps → one command, run every deploy.

Production Rollout: Canary, Monitoring, Rollback

Deploy Sequence

Gateway = blast radius max. Multi-step:

New cluster shadow validation : New gateway nodes, replay production logs — verify function & perf. Zero prod impact.

1% traffic cut : DNS/LB shift 1% to new cluster. Observe 24hrs — error rate, P99, rate-limit branch hits.

Gradual ramp : 10% → 30% → 50%, each ≥2hrs. Each step has explicit stop criteria: error rate >1.5× baseline, P99 > SLO, new error types.

Full cut : Keep old cluster 48hrs for instant rollback.

Decommission old : After confirmed stable, scale down old.

Must-Monitor Metrics (Four Categories)

Traffic : QPS (by Host, status), bandwidth, connection count.

Latency : request_time P50/P90/P99, upstream_response_time P99, delta (gateway self-cost).

Errors : 5xx rate, 4xx rate, upstream connect errors, timeouts.

Rate-limit : limit_result distribution ( :pass, :delay, :reject, :error), shared dict capacity %, ban count.

Three collection methods:

Log parsing : JSON access_log consumed by Filebeat/Vector → aggregation. Zero intrusion; minute-level latency.

log_by_lua push : Accumulate in shared dict, timer batch-push to monitoring. Real-time; requires code.

stub_status + custom exporter : ngx_http_stub_status_module for basic conn/req counts; Lua /metrics for custom.

Prometheus /metrics example (expose only on 127.0.0.1):

-- lua/admin/metrics.lua
local function emit(name, v) ngx.say(name, " ", v) end
function _M.run()
  ngx.header["Content-Type"] = "text/plain; version=0.0.4"
  local shm = ngx.shared.limit_req
  emit("gateway_shm_free_ratio", shm:free_space() / shm:capacity())
  emit("gateway_limit_reject_total", tonumber(ngx.shared.cfg_cache:get("stat:reject")) or 0)
end

Never expose /metrics publicly — contains internal state, no auth. listen 127.0.0.1:8080 + allow 127.0.0.1; deny all; double safety.

Change Risk Checklist

Upstream list : Traffic shifts to new nodes, may all hit cold nodes → Pre-check: new nodes pre-warmed, health checks pass → Rollback: revert list, reload.

Rate-limit threshold : Mass 429 or no-op → Pre-check: dry-run confirm blast radius → Rollback: admin API revert.

Shared dict size : Reload clears counters → instant traffic surge → Pre-check: assess backend headroom → Rollback: revert size, reload.

Auth logic : All 401 → Pre-check: full validation in staging → Rollback: revert code.

Lua module : Same + syntax errors → Pre-check: openresty -t + unit tests → Rollback: revert file, reload.

worker_processes : Brief connection drops → Pre-check: off-peak → Rollback: reload.

"Change shared dict size" most overlooked . lua_shared_dict allocated at startup; reload releases old, allocates new — all counters zeroed . If limit=100 QPS, dict clear → previously throttled traffic floods backend. Expand dict off-peak, or two-phase: add new dict, switch code, shrink old.

Permissions & Audit

Config files: owner root, group nginx, mode 640. Lua files same — never 777.

No plaintext secrets in config. Redis password, API key hashes via env vars or secret files.

Admin endpoints localhost-only + access logged. Who changed threshold, who pulled config — all traceable.

Changes require approval. Threshold/ban changes affect customers → audit trail. Implement via Git+CI: config change = PR, merge → auto-distribute to gateway dirs & reload.

Often Ignored: Clock Sync

Rate limiting, bans, logs all time-dependent. Multi-node clock drift causes:

Ban effective on Node A, Node B clock 5min slow → unban calculated early.

Log timestamps scrambled → causal chain broken during debugging.

Sliding window boundaries inconsistent → same IP different limit results per node.

Enforce NTP: chronyc tracking Output:

Reference ID  : A.B.C.D (ntp.example.com)
Stratum       : 2
System time   : 0.000123456 seconds fast of NTP time
Last offset   : +0.000098 seconds
RMS offset    : 0.000345 seconds
Frequency     : 12.345 ppm slow

System time

abs <10ms, RMS offset <1ms. Exceeds → investigate NTP source.

Appendix: Production-Ready Config Skeleton

Directory layout per §4.2.

11.1 conf.d/00-main.conf

# Log format
log_format gateway_json escape=json
'{"ts":"$time_iso8601","rid":"$request_id","host":"$host","uri":"$uri",'
'"method":"$request_method","status":$status,"bytes":$body_bytes_sent,'
'"rt":$request_time,"urt":"$upstream_response_time",'
'"uct":"$upstream_connect_time","uaddr":"$upstream_addr",'
'"ua":"$http_user_agent","ip":"$remote_addr","limit":"$limit_result"}';

access_log /var/log/nginx/access.log gateway_json buffer=64k flush=5s;

# Connection limit zone
limit_conn_zone $binary_remote_addr zone=perip_conn:20m;

# Health check upstream timeouts
proxy_connect_timeout 2s;
proxy_send_timeout 10s;
proxy_read_timeout 10s;
proxy_buffering on;
proxy_buffers 16 16k;
proxy_busy_buffers_size 32k;

# Hide version
server_tokens off;

# Request body/header limits (prevent large packet attacks)
client_max_body_size 8m;
client_header_buffer_size 4k;
large_client_header_buffers 4 16k;
client_body_timeout 15s;
client_header_timeout 15s;
send_timeout 30s;

client_header_buffer_size

+ large_client_header_buffers must match actual Cookie sizes. Default 1k returns 400 with request header too large — looks like client bug but is gateway config.

11.2 conf.d/90-admin.conf

server {
  listen 127.0.0.1:8080;
  server_name _;

  allow 127.0.0.1;
  deny all;

  access_log off;

  location = /_admin/config {
    content_by_lua_block { require("admin.config").dump() }
  }

  location = /_admin/metrics {
    content_by_lua_block { require("admin.metrics").run() }
  }

  location = /_admin/limit {
    # POST body: {"key":"1.2.3.4","rps":200}
    content_by_lua_block { require("admin.limit").apply() }
  }

  # Config center source, accessed via subrequest by lib.config
  location = /_internal/config {
    internal;  # ONLY internal subrequests
    proxy_pass http://127.0.0.1:9000/gateway/config;
    proxy_set_header Host $host;
    proxy_connect_timeout 1s;
    proxy_read_timeout 1s;
  }
}

internal

directive mandatory. Without it, external callers hit /_internal/config directly, exposing full config (whitelists, key hashes).

11.3 lua/init/worker.lua

local _M = {}
function _M.run()
  local cfg = require "lib.config"

  -- Config refresh: every 15s
  cfg.start_refresh(15)

  -- Metrics aggregation push: every 10s
  local ok, err = ngx.timer.every(10, function(premature)
    if premature then return end
    local ok2, err2 = pcall(function()
      require("lib.metrics").flush()
    end)
    if not ok2 then ngx.log(ngx.ERR, "metrics flush error: ", err2) end
  end)
  if not ok then ngx.log(ngx.ERR, "start metrics timer failed: ", err) end

  -- Banlist sync: every 30s
  ok, err = ngx.timer.every(30, function(premature)
    if premature then return end
    pcall(function() require("lib.banlist").sync() end)
  end)
  if not ok then ngx.log(ngx.ERR, "start banlist timer failed: ", err) end
end
return _M

Three timers, different periods by design: config fast (15s — quick effect), metrics moderate (10s — don't overload monitoring), banlist middle (30s — attacks persist, 30s delay acceptable).

11.4 Complete access_by_lua Orchestration

Order: Ban check → Auth → Rate limit → Pass. Benefit: banned requests consume zero compute.

-- lua/access/dispatcher.lua
local _M = {}
function _M.run()
  local waf   = require "access.waf"
  local auth  = require "access.auth"
  local limit = require "access.limit"

  -- 1) Ban check: shared dict only, cheapest
  if waf.check_ban() then return end

  -- 2) Auth: may hit Redis, only if credentials present
  auth.run()

  -- 3) Rate limit: IP then user, combined
  local ok, err = limit.run()
  if not ok then return end

  -- 4) Counting (for risk control)
  waf.count()
end
return _M

limit.run()

internally uses resty.limit.traffic.combine for IP+user atomic combo (§6.6).

11.5 Minimum Test Checklist (Run All Before Deploy)

1 No credentials → 401 →

curl -s -o /dev/null -w "%{http_code}" https://api.example.com/

2 Invalid credentials → 401 → curl -H "Authorization: Bearer bad" ... 3 Valid credentials → 200 → curl -H "Authorization: Bearer $KEY" ... 4 Exceed QPS threshold → 429 + Retry-After → wrk at 2× threshold

5 Concurrency exceed → 429 + limit=conn:reject → Slow endpoint concurrent hammer

6 Health check endpoint → 200, no rate-limit cost → 1000 rapid hits

7 Unknown Host → 502 fast → curl -H "Host: unknown.local" ... 8 Large Header → 400 not 502 → Craft 20KB Cookie

9 All upstreams down → 502 within 5s → Stop one upstream

10 During reload → Zero 5xx → Continuous wrk + 3 reloads

Case 10 most skipped but validates "deploy process itself causes no errors". Method: sustained wrk, 3 reloads mid-run, error rate must be 0.

11.6 APISIX / Kong Trade-offs

If you need exactly this (routing, auth, rate limit, observability), self-built OpenResty ~2000-3000 LOC, maintainable, fully controlled.

APISIX/Kong advantages: plugin ecosystem (rate limit, circuit breaker, OpenTelemetry ready), control plane (visual config, multi-tenant), community security updates. Costs: extra abstraction layer (debugging requires framework understanding), higher resource usage (APISIX needs etcd, Kong needs PostgreSQL), custom needs → write plugins (more indirect than raw Lua).

Decision rubric:

Team ≤5, stable requirements, latency-critical → self-build.

Need multi-tenant, self-service config, frequent new plugins → APISIX.

Already on Kong ecosystem (plugin marketplace) → stay Kong.

Avoid extremes. Seen self-built gateways hit 30k LOC; also seen teams write APISIX plugin just to add a header — both painful.

Closing

Gateway is unique: every request passes through, so any latency spike or false ban amplifies globally. Therefore gateway code must be more conservative than business code — fewer "clever" tricks, more instantly readable structures.

The most valuable takeaways aren't the algorithms but three disciplines:

Every Lua file: "Can this phase use this API?" — blocks half of production incidents.

Every rate-limit counter: "Who rolls back this increment?" — blocks the other half.

Every change: "How do I rollback, and what are rollback side-effects?" — decides if incident lasts 5 minutes or 5 hours.

Everything else is implementing these three disciplines in code and process.