Scaling Nginx to Handle 500M Daily Requests: From Reverse Proxy to Traffic Governance Hub

1. Business Background – Why the Simple Reverse‑Proxy Model Breaks

In a payment‑center scenario the daily request volume reaches about 5 × 10⁸, with normal peaks of 8 000 QPS and flash‑sale spikes of 100 000‑120 000 QPS. The original architecture consisted of App → SLB → Nginx → API Gateway → downstream services (order, payment, coupon, MySQL). Nginx only performed TLS termination, static file serving and proxy_pass.

Four typical failure modes appeared during a pre‑heat flash‑sale:

Non‑core traffic (activity pages, eligibility checks) crowded the core payment‑callback path, causing request queues and order‑status inconsistencies.

Hotspot data (coupon templates, product details) directly hit Java gateways and databases, inflating response times from milliseconds to seconds.

New order‑service versions were rolled out via full‑traffic switch; 30 % of abnormal traffic forced a whole‑system rollback, lacking fine‑grained gray control.

Nginx itself became a bottleneck: no connection reuse, caching or fine‑grained rate limiting caused CPU saturation and connection buildup.

The root cause is not a poorly written service but an edge layer that lacks traffic‑governance capabilities.

2. Why Nginx Is Suited for Edge Traffic Governance

Because Nginx sits before connection establishment, request parsing and routing, it can filter traffic earlier and save downstream resources.

Comparison of rejecting an over‑limit request:

At the Java gateway: the request has already passed TCP handshake, HTTP parsing, thread scheduling, object creation and routing – high cost.

At Nginx: rejection costs only a shared‑memory counter check and a few C‑level condition evaluations.

Suitable governance capabilities for Nginx include:

IP/URI/Header/Cookie based rate limiting and connection limiting.

URI/parameter based edge caching.

Header/Cookie/weight based gray release.

Upstream connection reuse, timeout, retry and passive circuit‑break.

Basic observability and log sampling.

Entry‑level degradation and unified error responses.

Capabilities that should stay out of Nginx:

Complex business‑level authentication orchestration.

Heavyweight workflow and transaction compensation.

Cross‑system distributed consistency.

Deep business‑object parsing.

3. Target Architecture – From Simple Proxy to Governance Cluster

The upgraded edge consists of a multi‑node Nginx cluster (Nginx‑1, Nginx‑2, Nginx‑3) that performs rate limiting, caching, gray routing and health‑check before forwarding to dedicated upstream pools (pay‑callback, activity, order, generic).

Four responsibilities are defined:

Protection : limit malicious, bursty or useless traffic, prioritize core paths (payment, callback, order).

Buffering : use local cache, stale cache and connection pools to absorb downstream spikes.

Splitting : route users, regions or versions to different backend pools for gray release and elastic scaling.

Fallback : when a pool fails or times out, apply timeout, retry, node removal and graceful degradation.

4. Rate Limiting – Why It Must Be Placed at Nginx

4.1 What Rate Limiting Protects

Beyond protecting downstream services, rate limiting also safeguards Nginx workers, upstream connection pools, gateway thread pools, Redis connections, DB connections and the payment‑callback chain.

4.2 Native Nginx Mechanism

Nginx uses two directives: limit_req_zone / limit_req – rate‑based limiting. limit_conn_zone / limit_conn – concurrent‑connection limiting.

Both rely on a shared‑memory zone that stores the last request timestamp, the current token‑bucket level and the active connection count.

4.3 Multi‑Dimensional Limits

Single‑IP limits are insufficient in corporate networks (NAT, CDN, shared IPs). A production‑grade configuration combines dimensions:

Client real IP.

User ID (after trusted authentication).

URI or API group.

Tenant.

Business priority.

4.4 Production‑Ready Configuration

worker_processes auto;
worker_rlimit_nofile 200000;

events {
    worker_connections 65535;
    multi_accept on;
    use epoll;
}

http {
    # limit per real IP – 120 req/s, 60 burst, no delay
    limit_req_zone $binary_remote_addr zone=per_ip:20m rate=120r/s;
    # limit per authenticated user – 30 req/s
    limit_req_zone $http_x_user_id zone=per_user:50m rate=30r/s;
    # map API to a limit key
    map $uri $api_limit_key {
        default "common";
        ~^/api/pay/callback "pay_callback";
        ~^/api/order/query   "order_query";
        ~^/api/activity/    "activity";
    }
    limit_req_zone $api_limit_key zone=per_api:20m rate=300r/s;

    limit_conn_zone $binary_remote_addr zone=conn_per_ip:20m;

    server {
        listen 443 ssl http2;
        server_name api.example.com;

        # payment callback – strict limit, fail‑close
        location = /api/pay/callback {
            limit_req   zone=per_ip burst=60 nodelay;
            limit_req   zone=per_api burst=120;
            limit_conn  conn_per_ip 80;
            proxy_pass  http://pay_callback_pool;
        }

        # activity – lower burst
        location ^~ /api/activity/ {
            limit_req   zone=per_ip burst=20 nodelay;
            limit_req   zone=per_api burst=30;
            proxy_pass  http://activity_pool;
        }

        # authenticated user APIs
        location ^~ /api/member/ {
            limit_req   zone=per_ip burst=40;
            limit_req   zone=per_user burst=10 nodelay;
            proxy_pass  http://gateway_pool;
        }

        # generic APIs
        location ^~ /api/ {
            limit_req   zone=per_ip burst=40;
            proxy_pass  http://gateway_pool;
        }

        limit_req_status 429;
        error_page 429 = @rate_limited;
        location @rate_limited {
            default_type application/json;
            add_header Retry-After 1 always;
            return 429 '{"code":429,"message":"Too many requests"}';
        }
    }
}

Key design points:

Use $binary_remote_addr instead of $remote_addr to reduce memory usage.

Separate thresholds for core and activity APIs to avoid priority mismatch.

Isolate authenticated and anonymous traffic into different limit buckets.

Return status 429 with a clear JSON body for downstream monitoring.

4.5 Estimating Limits

Example: 8 payment‑callback instances, each stable at 1 500 RPS, 20 % safety margin, 4 Nginx nodes → total target 9 600 RPS. Start testing per‑node limits at 2 000‑2 400 RPS and adjust based on real traffic distribution.

4.6 Single‑Node vs Distributed Limiting

Native limit_req works per node. With 4 nodes each limited to 100 r/s, the cluster caps at 400 r/s – sufficient for 80‑90 % of cases because the goal is to protect downstream, not to achieve mathematically exact global quotas.

Distributed limiting (Redis + Lua token bucket) is reserved for:

Critical interfaces needing near‑global precise quotas.

Multi‑region, multi‑LB traffic with severe imbalance.

Sample OpenResty + Redis Lua limiter:

local redis = require "resty.redis"
local red = redis:new()
red:set_timeout(50)
local ok, err = red:connect("redis-1.internal", 6379)
if not ok then ngx.log(ngx.ERR, "redis connect failed: ", err); return ngx.exit(503) end

local user_id = ngx.req.get_headers()["X-User-Id"] or "anonymous"
local key = "rl:pay:callback:" .. user_id
local limit = 20
local window = 1
local script = [[
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local current = redis.call("INCR", key)
if current == 1 then redis.call("EXPIRE", key, window) end
if current > limit then return 0 end
return current
]]
local res, eval_err = red:eval(script, 1, key, limit, window)
if not res then ngx.log(ngx.ERR, "redis eval failed: ", eval_err); return ngx.exit(503) end
if res == 0 then
    ngx.status = 429
    ngx.header["Content-Type"] = "application/json"
    ngx.say('{"code":429,"message":"rate limited"}')
    return ngx.exit(429)
end
red:set_keepalive(10000, 200)

Guidelines:

Native limiting has negligible overhead and no external dependency.

Distributed limiting adds a Redis round‑trip, requires Redis HA, and the limiter itself becomes a potential failure point.

5. Edge Caching – Making Nginx the First Read Defense

5.1 What Data Is Suitable for Edge Cache

Cache candidates satisfy at least one of:

Read‑heavy, write‑light.

Clearly hotspot.

Latency tolerance of seconds to minutes.

Backend query cost > cache miss cost.

Stale data acceptable when backend is down.

Typical examples: coupon templates, activity config, product basic details, region/store lists, non‑critical eligibility status.

5.2 How proxy_cache Works

Two components: keys_zone – in‑memory index of cached entries. proxy_cache_path – on‑disk storage of cached files.

Request flow:

Compute cache key.

Lookup keys_zone.

If hit, serve cached file.

If miss, fetch from upstream.

If response meets cache criteria, write to disk and register metadata.

Subsequent requests reuse the cached file.

5.3 Production‑Grade Cache Configuration

http {
    proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=edge_cache:512m max_size=50g inactive=30m use_temp_path=off;
    map $request_method $cacheable_method { default 0; GET 1; HEAD 1; }
    map $http_authorization $has_auth { default 1; "" 0; }
    map $arg_no_cache $force_bypass { default 0; 1 1; }

    server {
        listen 443 ssl http2;
        server_name api.example.com;

        location = /api/coupon/template {
            proxy_cache edge_cache;
            proxy_cache_methods GET HEAD;
            proxy_cache_key "$scheme|$proxy_host|$request_uri|$http_x_app_version";
            proxy_cache_valid 200 5m;
            proxy_cache_valid 301 302 10m;
            proxy_cache_valid 404 30s;
            proxy_cache_lock on;
            proxy_cache_lock_timeout 3s;
            proxy_cache_lock_age 5s;
            proxy_cache_background_update on;
            proxy_cache_revalidate on;
            proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
            proxy_no_cache $has_auth $force_bypass;
            proxy_cache_bypass $has_auth $force_bypass;
            add_header X-Cache-Status $upstream_cache_status always;
            add_header Cache-Control "public, max-age=300, stale-while-revalidate=30" always;
            proxy_pass http://coupon_pool;
        }
    }
}

Key design considerations:

Memory‑size of keys_zone must be sufficient; otherwise the index becomes a bottleneck.

Cache key must include all dimensions that affect response (scheme, host, URI, app version) to avoid wrong hits.

Use proxy_cache_lock to prevent cache stampede.

Enable proxy_cache_use_stale and background_update to serve stale data during backend failures.

5.4 Cache Consistency Strategy

Strong consistency is not realistic at the edge. Adopt a “cache‑aside invalidation” workflow:

Backend updates DB.

Backend clears Redis business cache.

Backend notifies edge cache via MQ or management API.

Next request fetches fresh data and repopulates the edge cache.

For strongly consistent data (payment status, inventory) disable edge caching entirely.

5.5 Cache Effectiveness Metrics

High hit rate alone is insufficient; monitor:

Back‑origin rate.

Stale‑hit proportion.

Hot‑key distribution.

Update‑to‑effective latency.

Cache‑error complaints.

6. Gray Release – Moving Beyond "Full Switch + Pray"

6.1 Why Nginx Is Ideal for Gray

Advantages:

Instant effect via nginx -s reload.

Routing decisions can be derived directly from Header, Cookie, URI, parameters.

Zero intrusion to Java processes.

Can be combined with rate limiting, caching and upstream health checks.

6.2 Production‑Ready Gray Configuration

upstream order_stable {
    server 10.0.0.101:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.102:8080 max_fails=3 fail_timeout=30s;
    keepalive 256;
}
upstream order_gray {
    server 10.0.1.101:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.102:8080 max_fails=3 fail_timeout=30s;
    keepalive 64;
}
# 1. Header whitelist
map $http_x_gray_version $gray_by_header { default 0; v2 1; }
# 2. Cookie based
map $cookie_canary $gray_by_cookie { default 0; 1 1; }
# 3. Region based
map $arg_area_id $gray_by_area { default 0; 330100 1; }
# 4. Percentage split (5%)
split_clients "$remote_addr$http_user_agent" $gray_percent {
    5% 1;
    * 0;
}
# Final decision
map "$gray_by_header$gray_by_cookie$gray_by_area$gray_percent" $order_upstream {
    default order_stable;
    ~1 order_gray;
}
server {
    listen 443 ssl http2;
    server_name api.example.com;
    location ^~ /api/order/ {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header X-Request-Id $request_id;
        proxy_set_header X-Gray-Upstream $order_upstream;
        proxy_pass http://$order_upstream;
    }
}

Design principles:

Whitelist users get gray traffic first.

Region‑specific validation via query param.

Small random traffic via split_clients for incremental rollout.

All rules collapse into a single variable for easy logging and troubleshooting.

6.3 Gray Release Process

Typical flow (illustrated in the article) includes monitoring the gray pool, observing business KPIs (payment success rate, order conversion), and rolling back instantly if anomalies appear.

7. High Availability – Beyond Two Nodes + VIP

7.1 Nginx HA with Keepalived

When a cloud‑managed L4 LB is not used, a classic VRRP solution provides fast VIP failover, health checks and split‑brain avoidance.

vrrp_script chk_nginx {
    script "/etc/keepalived/check_nginx.sh";
    interval 2;
    weight -20;
    fall 2;
    rise 1;
}

vrrp_instance VI_1 {
    state MASTER;
    interface eth0;
    virtual_router_id 51;
    priority 100;
    advert_int 1;
    authentication {
        auth_type PASS;
        auth_pass 1111;
    }
    virtual_ipaddress { 10.10.10.100/24; }
    track_script { chk_nginx; }
}

The check script validates both process existence and a local health endpoint to avoid false‑positive failover when workers are unresponsive.

7.2 Upstream HA in Open‑Source Nginx

Open‑source Nginx lacks active health checks; rely on passive mechanisms: max_fails + fail_timeout for automatic node removal. proxy_next_upstream to retry on defined errors. backup servers as fall‑back.

External platform periodically pushes updated upstream lists.

upstream pay_pool {
    least_conn;
    server 10.0.2.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.2.12:8080 max_fails=3 fail_timeout=30s;
    server 10.0.2.13:8080 backup;
    keepalive 512;
}
server {
    location = /api/pay/callback {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_connect_timeout 100ms;
        proxy_send_timeout 2s;
        proxy_read_timeout 2s;
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
        proxy_next_upstream_timeout 500ms;
        proxy_pass http://pay_pool;
    }
}

7.3 Connection Reuse and TIME_WAIT

Enabling upstream keepalive pools and clearing the Connection header prevents excessive TIME_WAIT accumulation and port exhaustion.

upstream gateway_pool {
    least_conn;
    server 10.0.3.11:8080;
    server 10.0.3.12:8080;
    keepalive 1024;
}
location /api/ {
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Request-Id $request_id;
    proxy_pass http://gateway_pool;
}

7.4 Retry Discipline

Uncontrolled retries amplify traffic. Recommendations:

Read requests: allow a few retries.

Write requests: retry only on network errors and only if the operation is idempotent.

Critical payment callbacks: prefer fail‑close or limited retry, leaving idempotent compensation to the business layer.

8. Scalability & Capacity Modeling

Key principles for high‑concurrency edge design:

Stateless nodes – only cache and logs are local.

Horizontal scaling – capacity grows linearly with node count.

Configuration layering – static Nginx config separates from dynamic policy (maps, limits).

Enterprise topology example includes CDN/WAF, L4 LB, multiple Nginx edge nodes, gateway clusters, Redis, MQ, sharded MySQL, illustrating that Nginx remains a pure traffic‑governance layer.

Separate upstream pools per business domain (pay‑callback, activity, order, generic) to achieve independent connection pools, timeouts, limits and gray routing.

Integration with asynchronous processing (MQ) allows the edge to smooth spikes while backend handles eventual consistency.

9. Observability – Metrics, Logging and Alerting

Essential Nginx‑level metrics:

Total and per‑API request counts.

2xx/4xx/5xx ratios.

Rate‑limit hit counts.

Upstream timeout and retry counts.

Cache hit/miss/stale ratios.

Upstream response time (avg, P95, P99).

Active/idle connection counts.

Gray‑pool vs stable‑pool traffic split.

Structured JSON log format (example from baseline):

log_format main_ext escape=json '{'
    "time":"$time_iso8601",
    "remote_addr":"$remote_addr",
    "real_ip":"$http_x_forwarded_for",
    "request_id":"$request_id",
    "host":"$host",
    "method":"$request_method",
    "uri":"$request_uri",
    "status":$status,
    "body_bytes_sent":$body_bytes_sent,
    "request_time":$request_time,
    "upstream_addr":"$upstream_addr",
    "upstream_status":"$upstream_status",
    "upstream_response_time":"$upstream_response_time",
    "cache_status":"$upstream_cache_status"'
'}';
access_log /var/log/nginx/access.log main_ext buffer=256k flush=1s;

Key fields for troubleshooting: request_id, real_ip, request_uri, status, request_time, upstream_addr, upstream_status, upstream_response_time, upstream_cache_status.

Alerting should go beyond 5xx, e.g., sudden rise in core‑API rate‑limit hits, cache MISS spikes, upstream RT increase without 5xx, abnormal gray‑pool traffic share, or high active connections on a single node.

10. Security Considerations

10.1 Trusted Real‑IP Restoration

Never trust arbitrary X-Forwarded-For. Configure set_real_ip_from with only the CIDR blocks of your CDN/LB.

10.2 Management Interface Isolation

Endpoints that trigger cache invalidation, hot‑reload or gray switches must be reachable only from internal network, protected by mTLS or IP whitelist, and emit audit logs.

10.3 Sensitive Data Redaction

Avoid logging raw phone numbers, ID numbers, tokens, payment credentials or full cookie contents. Apply masking or omit these fields entirely.

11. Common Pitfalls & Checklist

Treating Nginx as a full‑featured application gateway – keep business logic out of Lua/config.

Using a single timeout/retry profile for all APIs – tailor per‑service.

Only request‑rate limiting without connection limiting – expose to slow‑loris attacks.

Enabling cache without proper key design – leads to wrong‑hits.

Relying solely on cache hit ratio – monitor miss, stale and back‑origin rates.

Gray releases evaluated only by HTTP 5xx – also watch business KPIs (payment success, coupon redemption, regional conversion).

12. Evolution Roadmap

Dynamic Config Center : externalize limits, gray percentages, upstream tags via etcd/OpenResty or a config‑generation platform.

Multi‑Region Multi‑Active : DNS/GSLB proximity routing, same‑city dual‑active, cross‑region active‑active deployments.

Service Mesh Collaboration : keep Nginx for external ingress, WAF/CDN, TLS termination, edge cache, while delegating intra‑service traffic governance to Envoy or Istio.

Platformization : configuration UI, real‑time metrics dashboard, one‑click rollback, automated safety checks.

13. Prioritized Implementation Phases

Phase 1 – Core Foundations : real‑IP restoration, upstream keepalive, per‑API timeouts, structured JSON logging, basic limit_req and limit_conn.

Phase 2 – Edge Caching : enable proxy_cache for public APIs, lock to prevent stampede, use_stale for fault tolerance, fine‑tune cache keys.

Phase 3 – Gray Release & HA : map‑based gray routing, upstream pool segregation, Keepalived VIP failover, upstream passive health checks.

Phase 4 – Platformization : dynamic limit store, multi‑region deployment, integration with service mesh and CI/CD release pipelines.

14. Final Takeaway

When Nginx is elevated from a mere request forwarder to the system’s first line of traffic governance – handling rate limiting, edge caching, gray releases, high availability and observability – many problems that would otherwise require heavyweight application changes are solved at lower cost and earlier in the request path. This transformation is essential for any high‑throughput, latency‑sensitive enterprise service.