Production Nginx for 10M+ QPS: Rate Limiting, Load Balancing, WebSocket, gRPC, CDN

Why Many Teams Still Fail Under High Traffic

Even when a system runs smoothly under normal load, sudden spikes from promotions, live events, short‑video redirects, or client retry storms can cause chain‑level failures such as:

No rate limiting at the entry layer, instantly saturating the application thread pool.

Insufficient upstream connection pool, leading to a surge of TIME_WAIT sockets.

Poor upstream load‑balancing configuration, sending traffic to slow nodes.

No cache hierarchy for static assets or hot endpoints, causing cache‑thundering directly to the database.

WebSocket and HTTP API sharing the same timeout parameters, causing frequent long‑connection rebuilds.

gRPC multiplexed connections creating traffic skew that overloads individual nodes.

Nginx lacks observable metrics, leaving failures to be guessed.

In reality, the problem often lies in the entry layer not fulfilling four core responsibilities:

Traffic shaping: turning bursty spikes into a steady, consumable flow.

Connection reuse: consolidating massive short connections at the edge.

Cache offload: keeping repeated requests out of the business system.

Fault isolation: containing local anomalies within the entry or service group.

Nginx’s Real Position in a High‑Concurrency Architecture

For high‑traffic systems, Nginx is not merely a reverse‑proxy; it is the traffic‑governance hub. A typical internet entry chain looks like:

Client → CDN / WAF / Anti‑DDoS → L4 SLB / LVS / CLB → Nginx Edge Cluster → Nginx Service Gateway → Application Services (HTTP, gRPC, WebSocket) → Redis / MQ / DB / Search

Within this chain, Nginx handles Layer‑7 duties such as TLS termination, protocol adaptation, unified entry for HTTP/1.1, HTTP/2, WebSocket, gRPC, routing (URI, header, cookie, gray release), rate limiting, black‑/white‑listing, authentication, static file serving, edge caching, origin convergence, fault isolation, fast upstream failover, and observability (metrics, log passthrough).

A production‑grade Nginx solution must be evaluated on:

Stable support for high‑concurrency connections.

Protection of downstream services under abnormal traffic.

Ability to scale smoothly and apply dynamic governance.

Continuous observability, tuning, gray‑release, and audit capability.

Underlying Principles: Why Nginx Handles High Concurrency

3.1 Multi‑Process + Event‑Driven Model

Nginx uses a master + worker architecture: master reads configuration, manages workers, handles signals. worker processes actual requests.

Each worker runs a single‑threaded event loop based on epoll (or other I/O multiplexing), handling massive connections without per‑request threads.

This design yields stable memory usage, minimal context switches, and high CPU efficiency—ideal for short‑lived requests, static resources, reverse proxying, and long‑connection forwarding.

3.2 High‑Concurrency Capacity Estimation

The theoretical maximum concurrent connections are roughly:

theoretical_max_conns ≈ worker_processes * worker_connections

Real usable concurrency is limited by:

System file‑descriptor limit ( ulimit -n).

One front‑end connection often maps to one upstream connection.

Keep‑alive pool consumption.

Access log, cache files, temporary files using descriptors.

Network stack limits (backlog, port range, conntrack capacity).

Practical capacity is the minimum of connection capacity, CPU processing power, network throughput, upstream service capacity, and cache hit capability.

3.3 Request Lifecycle

Accept connection.

Parse request line and headers.

Match server and location.

Execute rewrite / access / limit / auth phases.

If static file, return directly.

If proxy, select upstream node.

Reuse or create upstream connection.

Read response, decide cache/compression/logging.

Return to client.

Understanding this flow is crucial because many misconfigurations stem from placing directives in the wrong phase.

Production Baseline Configuration

user  nginx;
worker_processes  auto;
worker_rlimit_nofile 200000;

error_log  /var/log/nginx/error.log  warn;
pid        /var/run/nginx.pid;

events {
    use epoll;
    worker_connections 65535;
    multi_accept on;
    accept_mutex off;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    server_tokens off;
    charset        utf-8;

    sendfile        on;
    tcp_nopush      on;
    tcp_nodelay     on;
    aio             threads;
    directio        4m;

    keepalive_timeout  65;
    keepalive_requests 10000;
    reset_timedout_connection on;

    client_header_timeout 10s;
    client_body_timeout   15s;
    client_max_body_size  64m;
    send_timeout           15s;

    types_hash_max_size   4096;
    client_body_buffer_size 256k;
    client_header_buffer_size 4k;
    large_client_header_buffers 8 32k;

    resolver 127.0.0.1 valid=10s ipv6=off;
    resolver_timeout 3s;

    gzip on;
    gzip_comp_level 5;
    gzip_min_length 1k;
    gzip_types text/plain text/css application/json application/javascript application/xml text/xml image/svg+xml;

    log_format json_main escape=json '{"time":"$time_iso8601","remote_addr":"$remote_addr","request_id":"$request_id","host":"$host","uri":"$uri","args":"$args","request_method":"$request_method","status":$status,"body_bytes_sent":$body_bytes_sent,"request_time":$request_time,"upstream_connect_time":"$upstream_connect_time","upstream_header_time":"$upstream_header_time","upstream_response_time":"$upstream_response_time","upstream_addr":"$upstream_addr","upstream_status":"$upstream_status","http_referer":"$http_referer","http_user_agent":"$http_user_agent","xff":"$http_x_forwarded_for","cache_status":"$upstream_cache_status"}';
    access_log /var/log/nginx/access.log json_main;

    include /etc/nginx/conf.d/*.conf;
}

4.1 Engineering Implications

worker_processes auto

usually matches CPU cores. worker_rlimit_nofile 200000 must be coordinated with the system ulimit -n. multi_accept on reduces accept overhead during connection storms. accept_mutex off is generally better on modern kernels with multi‑queue NICs, but should be validated with load tests. aio threads + sendfile improves static and large‑file delivery. keepalive_requests 10000 avoids frequent connection teardown. reset_timedout_connection on quickly recovers timed‑out connections. large_client_header_buffers prevents 400 errors from oversized cookies or auth headers.

JSON logs enable seamless ingestion into ELK, ClickHouse, Loki, or Kafka.

4.2 Kernel Parameter Recommendations

fs.file-max = 1000000
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.netfilter.nf_conntrack_max = 1048576

In high‑connection scenarios, Nginx, the kernel, and the application must be tuned together; otherwise you may see silent client‑side latency spikes, handshake failures, or a surge of 502 errors.

Production‑Grade Reverse Proxy Design

5.1 Upstream Strategy Selection

round_robin

: default, suitable for stateless HTTP services. least_conn: prefers nodes with the fewest active connections; better for long‑running or slow backends (Java, Go, Node). ip_hash: sticky routing based on client IP; useful for weak session affinity. hash $key consistent: consistent hashing for cache‑sharding or session‑affinity scenarios.

upstream api_backend {
    least_conn;
    server 10.10.1.11:8080 max_fails=3 fail_timeout=10s;
    server 10.10.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.10.1.13:8080 max_fails=3 fail_timeout=10s;
    keepalive 512;
}

Key points: keepalive manages the pool of connections from Nginx to upstream, not client connections. max_fails + fail_timeout provides passive failure removal; it does not replace active health checks.

For Java/Go/Node services that may experience occasional Full GC or event‑loop stalls, least_conn often yields more stable performance than simple round‑robin.

5.2 Real‑IP Propagation and Tracing

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Request-Id $request_id;
proxy_set_header X-Forwarded-Host $host;

If an upstream SLB/CDN/WAF sits in front, combine with:

set_real_ip_from 10.0.0.0/8;
set_real_ip_from 172.16.0.0/12;
set_real_ip_from 192.168.0.0/16;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

Otherwise $remote_addr will only show the previous hop, distorting rate‑limiting and audit logs.

5.3 Production‑Grade API Proxy Template

server {
    listen 80 reuseport;
    server_name api.example.com;

    location / {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Request-Id $request_id;

        proxy_connect_timeout 2s;
        proxy_send_timeout 8s;
        proxy_read_timeout 8s;
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
        proxy_pass http://api_backend;
    }
}

5.4 Common Pitfalls in API Proxy

Do not forward Connection: keep-alive to upstream; use proxy_set_header Connection "" to strip hop‑by‑hop headers.

Avoid overly large proxy_connect_timeout as it delays failure recovery.

Limit proxy_next_upstream_tries to prevent retry storms.

Enable proxy_buffering for typical HTTP APIs to improve throughput; disable for streaming scenarios.

Engineering Rate Limiting

6.1 Rate‑Limiting Fundamentals

Nginx implements a leaky‑bucket/token‑bucket approximation. It:

Consumes requests at a fixed rate.

Allows a burst of requests to queue.

Rejects only when the queue overflows.

limit_req_zone $binary_remote_addr zone=per_ip_rate:20m rate=50r/s;
limit_req_zone $server_name zone=global_rate:10m rate=20000r/s;

Example usage for an order‑creation endpoint:

location /api/order/create {
    limit_req zone=global_rate burst=5000 nodelay;
    limit_req zone=per_ip_rate burst=20 nodelay;
    proxy_pass http://api_backend;
}

Site‑wide ceiling: 20 000 r/s.

Per‑IP smooth rate: 50 r/s with a burst of 20. nodelay lets burst traffic be processed immediately instead of queuing.

6.2 Why Global IP Limiting Is Insufficient

In CDN, SLB, carrier NAT, or mobile networks, a single user may appear under many IPs, and many users may share one NAT IP. Therefore, limiting solely by $binary_remote_addr can over‑ or under‑limit. A better key combines business identifiers:

map $http_x_user_id $limit_uid {
    default $http_x_user_id;
    ""      $binary_remote_addr;
}
limit_req_zone $limit_uid zone=user_rate:50m rate=100r/s;

Similar keys can be built from app ID, tenant ID, channel ID, token, or device ID.

6.3 Connection‑Concurrency Limiting

Rate limiting does not protect against slow‑connection attacks or idle long‑connections. Combine with limit_conn:

limit_conn_zone $binary_remote_addr zone=per_ip_conn:20m;
server {
    location / {
        limit_conn per_ip_conn 30;
        limit_conn_status 429;
        proxy_pass http://api_backend;
    }
}

6.4 Hierarchical Limiting Example

Three‑tier protection: /health and /metrics: internal whitelist, no limiting. /api/order/create: strict limiting to protect inventory services. /api/search: larger burst because the endpoint is cacheable. /callback/payment: whitelist for payment callbacks. /ws/: limit concurrent connections rather than request rate.

geo $is_internal_ip {
    default 0;
    10.0.0.0/8 1;
    172.16.0.0/12 1;
    192.168.0.0/16 1;
}
map $is_internal_ip $skip_limit {
    1 1;
    0 0;
}
limit_req_zone $binary_remote_addr zone=write_api_rate:20m rate=30r/s;
limit_req_zone $binary_remote_addr zone=read_api_rate:20m rate=200r/s;
server {
    location /health { access_log off; return 200 "ok
"; }
    location /api/order/create {
        if ($skip_limit = 0) { limit_req zone=write_api_rate burst=20 nodelay; }
        proxy_pass http://api_backend;
    }
    location /api/search {
        if ($skip_limit = 0) { limit_req zone=read_api_rate burst=100 nodelay; }
        proxy_pass http://api_backend;
    }
}

Note: if should be used cautiously; map ‑based gating is preferred for production.

Advanced Load Balancing

7.1 Limitations of Passive Health Checks

Open‑source Nginx only removes nodes after failed responses. It cannot detect:

Port reachable but thread pool exhausted.

200 responses with high latency.

Full GC pauses causing intermittent timeouts.

Production‑grade setups therefore add:

Application‑level /healthz and /readyz endpoints.

External SLB/K8s sidecar performing active probing.

Nginx focuses on fast failover, leaving slow‑node eviction to the control plane.

7.2 Slow‑Start & Node Warm‑Up

New nodes should receive a small traffic fraction initially (5 % → 20 % → 50 % → 100 %) and be pre‑warmed (JIT, caches, connection pools). Open‑source Nginx lacks built‑in slow‑start; common work‑arounds include adjusting upstream weight via SLB, service‑registry weight changes, or using API‑gateway meshes (APISIX, Kong, Envoy).

7.3 Consistent Hashing Use‑Case

When backends are distributed caches, session shards, or image‑processing nodes, consistent hashing improves cache hit rates and reduces data migration on node changes:

upstream profile_backend {
    hash $cookie_user_token consistent;
    server 10.10.2.11:8080;
    server 10.10.2.12:8080;
    server 10.10.2.13:8080;
}

Drawbacks: not suitable for highly heterogeneous node capacities; hot users may concentrate traffic.

WebSocket Long‑Connection Governance

8.1 Standard Proxy Configuration

upstream ws_backend {
    least_conn;
    server 10.20.1.11:9001 max_fails=3 fail_timeout=10s;
    server 10.20.1.12:9001 max_fails=3 fail_timeout=10s;
    keepalive 256;
}
server {
    listen 80;
    server_name ws.example.com;
    location /ws/ {
        proxy_pass http://ws_backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_read_timeout 1800s;
        proxy_send_timeout 1800s;
        proxy_connect_timeout 3s;
        proxy_buffering off;
    }
}

8.2 Why WebSocket Should Not Share API Parameters

API focuses on request throughput and short‑connection RTT.

WebSocket cares about total connections, heartbeats, reconnection, and session affinity.

Mixing parameters leads to short proxy_read_timeout causing disconnections, insufficient keepalive or worker_connections capacity, and lingering old workers after reload.

8.3 Best Practices

Deploy WebSocket on a dedicated domain or Nginx cluster.

Separate monitoring, capacity models, connection pools, and release cadence from ordinary API.

Apply connection‑count limits and heartbeat reclamation specifically for WebSocket.

8.4 Reload vs Long‑Connection Conflict

During nginx -s reload, new workers start while old workers wait for existing connections to close. Persistent WebSocket connections prevent old workers from exiting, leading to memory and FD leaks. Mitigation:

Reasonable client‑side heartbeat timeout.

Graceful shutdown: drain traffic, wait for idle connections, then reload.

K8s preStop hook and terminationGracePeriodSeconds for orderly termination.

Use dedicated ingress for long‑lived connections.

8.5 Real‑World IM Push Gateway Example

Assume an IM gateway must hold 200 k concurrent connections. Typical Nginx settings:

worker_processes=8

worker_connections=65535

Even though the theoretical connection capacity is sufficient, you must consider FD limits, memory, and heartbeat traffic. Randomizing client heartbeat intervals and decoupling the WebSocket layer from the message‑delivery layer (e.g., using MQ) dramatically reduces spikes.

gRPC Gateway Implementation

9.1 Standard gRPC Proxy

upstream grpc_backend {
    least_conn;
    server 10.30.1.11:50051 max_fails=2 fail_timeout=5s;
    server 10.30.1.12:50051 max_fails=2 fail_timeout=5s;
    keepalive 128;
}
server {
    listen 443 ssl http2;
    server_name grpc.example.com;
    ssl_certificate /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/privkey.pem;
    location / {
        grpc_pass grpc://grpc_backend;
        grpc_set_header Host $host;
        grpc_set_header X-Real-IP $remote_addr;
        grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        grpc_set_header X-Request-Id $request_id;
        grpc_connect_timeout 2s;
        grpc_read_timeout 60s;
        grpc_send_timeout 60s;
        grpc_next_upstream error timeout http_502 http_503 http_504;
        grpc_next_upstream_tries 2;
    }
}

9.2 Common gRPC Pitfalls

Pitfall 1: Long‑connection multiplexing can cause traffic skew. Use least_conn or limit connection lifetimes.

Pitfall 2: Streaming RPCs need longer timeouts than short calls (3‑10 s for unary, minutes for bidirectional streams).

Pitfall 3: HTTP error codes (502/503/504) do not map directly to gRPC status codes; unify error semantics to avoid client misinterpretation.

9.3 gRPC Health‑Check Recommendation

Implement the gRPC Health Checking Protocol in services, let SLB/K8s perform active probing, and let Nginx act only as a fast‑fail proxy.

Cache System Upgrade

10.1 Multi‑Level Cache Architecture

Four‑tier cache hierarchy for high‑traffic sites:

User → Browser Cache → CDN Edge Cache → Nginx proxy_cache → App Local Cache / Redis → DB / Search / Upstream API

Browser cache reduces duplicate downloads.

CDN edge absorbs nationwide traffic.

Nginx second‑level cache mitigates origin load, prevents cache‑thundering and snow‑ball effects.

Application‑level cache / Redis reduces DB reads.

10.2 Static Resource Service Template

server {
    listen 80;
    server_name static.example.com;
    location /assets/ {
        alias /data/www/assets/;
        expires 30d;
        add_header Cache-Control "public, max-age=2592000, immutable";
        access_log off;
        sendfile on;
        tcp_nopush on;
        open_file_cache max=200000 inactive=60s;
        open_file_cache_valid 120s;
        open_file_cache_min_uses 2;
        open_file_cache_errors on;
    }
}

Key points: use hashed filenames (e.g., app.3fd9b7.js) and immutable to let browsers cache forever; open_file_cache reduces filesystem metadata overhead.

10.3 proxy_cache Production Settings

proxy_cache_path /var/cache/nginx/proxy levels=1:2 keys_zone=api_cache:512m max_size=100g inactive=30m use_temp_path=off;
map $request_method $cache_bypass_by_method {
    default 1;
    GET 0;
    HEAD 0;
}
server {
    listen 80;
    server_name page.example.com;
    location /api/content/ {
        proxy_cache api_cache;
        proxy_cache_key "$scheme$request_method$host$request_uri";
        proxy_cache_methods GET HEAD;
        proxy_cache_valid 200 1m;
        proxy_cache_valid 404 10s;
        proxy_cache_lock on;
        proxy_cache_lock_timeout 5s;
        proxy_cache_background_update on;
        proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;
        proxy_no_cache $cache_bypass_by_method;
        proxy_cache_bypass $http_cache_control $arg_nocache;
        add_header X-Cache-Status $upstream_cache_status always;
        add_header Cache-Control "public, max-age=60";
        proxy_pass http://api_backend;
    }
}

10.4 Dealing with Cache Misses, Penetration, and Snowball

Cache Miss (Thundering Herd): enable proxy_cache_lock on, keep hot keys from expiring, use async background refresh or application‑level singleflight.

Cache Penetration: short‑TTL 404, validate parameters early, employ Bloom filters.

Cache Snowball: add random TTL jitter, use multi‑level caches, serve stale data, combine with rate limiting and graceful degradation.

10.5 Case Study – Product Detail Page During Flash Sale

Typical characteristics: massive GET traffic, low update frequency, tolerance for second‑level cache latency. Recommended stack:

CDN cache 30 s.

Nginx proxy_cache 10 s.

Redis cache 5‑30 s.

Backend DB receives only a tiny fraction of traffic.

This layered approach prevents the database from being overwhelmed even when traffic spikes hundreds of times.

Dynamic Configuration & Service Discovery

11.1 Dynamic DNS Resolution

resolver kube-dns.kube-system.svc.cluster.local valid=5s ipv6=off;
server {
    location / {
        set $backend_host "order-service.default.svc.cluster.local";
        proxy_pass http://$backend_host:8080;
    }
}

Pros: simplicity. Cons: loses upstream advanced features, adds DNS lookup overhead, limited governance.

11.2 Centralized Configuration Solutions

Common approaches for large fleets:

Nginx + Consul/Nacos/Etcd to generate upstream blocks dynamically.

OpenResty + Lua to pull configs at runtime.

Control‑plane gateways like Kong, APISIX, or Envoy.

Kubernetes Ingress Controller or Gateway API.

When services reach dozens or hundreds, manual upstream maintenance becomes untenable.

11.3 Hot‑Update Release Requirements

Production‑grade dynamic config should follow a verified, gray, rollback‑able workflow:

Configuration change audit.

Syntax check ( nginx -t).

Gradual rollout (canary).

Metric observation.

Automatic rollback on anomalies.

Containerization & Kubernetes

12.1 Secure Dockerfile

FROM nginx:1.25-alpine
COPY nginx.conf /etc/nginx/nginx.conf
COPY conf.d/ /etc/nginx/conf.d/
COPY html/ /usr/share/nginx/html/
RUN mkdir -p /var/cache/nginx /var/run /var/log/nginx \
    && chown -R nginx:nginx /var/cache/nginx /var/run /var/log/nginx /etc/nginx /usr/share/nginx/html
USER nginx
EXPOSE 8080 8443
STOPSIGNAL SIGQUIT
CMD ["nginx", "-g", "daemon off;"]

Key practices: non‑root user, explicit cache/log directories, graceful stop signal, avoid embedding certificates or secrets.

12.2 Kubernetes Deployment Snippet

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-edge
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: nginx-edge
  template:
    metadata:
      labels:
        app: nginx-edge
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: nginx
          image: registry.example.com/nginx-edge:1.0.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 3
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "nginx -s quit; sleep 15"]
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "2"
              memory: "2Gi"

12.3 Common K8s Issues & Remedies

Issue 1 – Reload causing connection jitter: Use maxUnavailable: 0, proper readinessProbe, and preStop to drain connections before pod termination.

Issue 2 – ConfigMap updates not applied: Ensure file sync, syntax validation, and explicit nginx -s reload (or sidecar/operator) after ConfigMap change.

Issue 3 – Overlapping responsibilities with Ingress Controller: Avoid duplicating edge‑level Nginx logic inside business pods when an Ingress Controller already provides it.

Observability

13.1 Mandatory Access Log Fields

Include at least: timestamp, request ID, client IP, host/URI/method, status, request latency, upstream address, upstream status, upstream response time, cache status, X‑Forwarded‑For.

13.2 Key Prometheus Metrics

nginx_connections_active

nginx_connections_reading

nginx_connections_writing

nginx_connections_waiting

nginx_http_requests_total

5xx error ratio

499 error ratio upstream_response_time, upstream_connect_time Cache hit ratio

TLS handshake failure rate

13.3 Why 499 Matters

Code 499 indicates client‑initiated connection close. It can signal slow responses, aggressive front‑end timeouts, user cancellations, mobile network instability, or mis‑configured rate‑limit back‑off strategies.

13.4 Alerting Recommendations

5‑minute 5xx ratio exceeds threshold.

Upstream connect timeout trends upward.

Cache hit rate drops sharply.

Active connections approach configured limits.

Reload frequency spikes.

TLS certificate expiration alerts.

Production Fault‑Troubleshooting Handbook

14.1 502 Bad Gateway

Typical causes: upstream process crash, unreachable upstream port, Nginx‑to‑upstream timeout, malformed upstream response headers.

Investigation steps:

Check Nginx error log.

Inspect upstream address and health status.

From the Nginx node, curl the upstream directly.

Examine application thread pool, GC pauses, connection pool saturation.

14.2 504 Gateway Timeout

Typical causes: backend processing too long, insufficient proxy_read_timeout, downstream DB or Redis latency spikes.

Usually the bottleneck is upstream, not Nginx.

14.3 High‑Peak RT Spike with Low CPU

Common reasons: upstream connection pool exhaustion, DNS resolution delays, disk I/O slowdown for cache files, proxy_cache_lock queuing.

Correlate RT with upstream connect time, active connections, disk utilization, and FD usage.

14.4 "Too many open files"

Fix by adjusting: worker_rlimit_nofile System ulimit -n Container runtime FD limits

Keepalive pool size

WebSocket long‑connection count

14.5 Reload Leaves Old Workers Running

Root causes: lingering WebSocket connections, unfinished large file downloads, ongoing gRPC streams.

Solution: design business‑level graceful termination, not just infinite reload windows.

Full Production Example – E‑Commerce Flash‑Sale Gateway

The following consolidated configuration demonstrates how to wire together API, WebSocket, gRPC, static assets, rate limiting, caching, and observability for a large‑scale e‑commerce platform.

15.1 Upstream Definitions

upstream api_backend {
    least_conn;
    server 10.10.1.11:8080 max_fails=3 fail_timeout=10s;
    server 10.10.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.10.1.13:8080 max_fails=3 fail_timeout=10s;
    keepalive 512;
}
upstream ws_backend {
    hash $http_x_user_id consistent;
    server 10.20.1.11:9001 max_fails=3 fail_timeout=10s;
    server 10.20.1.12:9001 max_fails=3 fail_timeout=10s;
    keepalive 256;
}
upstream grpc_backend {
    least_conn;
    server 10.30.1.11:50051 max_fails=2 fail_timeout=5s;
    server 10.30.1.12:50051 max_fails=2 fail_timeout=5s;
    keepalive 128;
}

15.2 Rate‑Limiting & Cache Zones

limit_req_zone $binary_remote_addr zone=common_ip_rate:20m rate=100r/s;
limit_req_zone $http_x_user_id zone=user_write_rate:50m rate=20r/s;
limit_conn_zone $binary_remote_addr zone=common_ip_conn:20m;
proxy_cache_path /var/cache/nginx/proxy levels=1:2 keys_zone=page_cache:512m max_size=100g inactive=30m use_temp_path=off;

15.3 API Service

server {
    listen 80 reuseport;
    server_name api.example.com;
    location /health { access_log off; return 200 "ok
"; }
    location /api/order/create {
        limit_req zone=common_ip_rate burst=20 nodelay;
        limit_req zone=user_write_rate burst=10 nodelay;
        limit_conn common_ip_conn 20;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Request-Id $request_id;
        proxy_connect_timeout 2s;
        proxy_send_timeout 8s;
        proxy_read_timeout 8s;
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
        proxy_pass http://api_backend;
    }
    location /api/content/ {
        limit_req zone=common_ip_rate burst=50 nodelay;
        proxy_cache page_cache;
        proxy_cache_key "$scheme$request_method$host$request_uri";
        proxy_cache_valid 200 30s;
        proxy_cache_valid 404 10s;
        proxy_cache_lock on;
        proxy_cache_background_update on;
        proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
        add_header X-Cache-Status $upstream_cache_status always;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Request-Id $request_id;
        proxy_pass http://api_backend;
    }
}

15.4 WebSocket Service

server {
    listen 80 reuseport;
    server_name ws.example.com;
    location /ws/ {
        limit_conn common_ip_conn 10;
        proxy_pass http://ws_backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-User-Id $http_x_user_id;
        proxy_connect_timeout 3s;
        proxy_read_timeout 1800s;
        proxy_send_timeout 1800s;
        proxy_buffering off;
    }
}

15.5 gRPC Service

server {
    listen 443 ssl http2;
    server_name grpc.example.com;
    ssl_certificate /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/privkey.pem;
    location / {
        grpc_pass grpc://grpc_backend;
        grpc_set_header Host $host;
        grpc_set_header X-Real-IP $remote_addr;
        grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        grpc_set_header X-Request-Id $request_id;
        grpc_connect_timeout 2s;
        grpc_send_timeout 30s;
        grpc_read_timeout 30s;
        grpc_next_upstream error timeout http_502 http_503 http_504;
        grpc_next_upstream_tries 2;
    }
}

15.6 Static Asset Service

server {
    listen 80 reuseport;
    server_name static.example.com;
    location /assets/ {
        alias /data/www/assets/;
        expires 30d;
        add_header Cache-Control "public, max-age=2592000, immutable";
        access_log off;
        sendfile on;
        tcp_nopush on;
        open_file_cache max=200000 inactive=60s;
        open_file_cache_valid 120s;
        open_file_cache_min_uses 2;
        open_file_cache_errors on;
    }
}

This configuration demonstrates independent governance strategies for each traffic type: strict protection for write APIs, aggressive caching for reads, dedicated connection handling for WebSocket, timeout tuning for gRPC, and bandwidth‑optimized static serving.

When to Stay with Nginx vs. Upgrade to API Gateway or Service Mesh

16.1 Continue Using Nginx

Service count under a few dozen.

Routing rules relatively stable.

Primary needs: high‑performance reverse proxy, TLS termination, caching, simple rate limiting.

Ops team already proficient with Nginx.

16.2 Upgrade to API Gateway

Need plugin‑based auth, signature verification, tenant isolation, quota management.

Require dynamic routing, gray releases, traffic mirroring.

Self‑service operation for multiple teams.

Centralized control plane for large‑scale configuration.

Typical choices: Kong, APISIX, custom OpenResty gateway.

16.3 Adopt Service Mesh

Complex east‑west traffic governance.

Need mTLS, circuit breaking, retries, standardized observability.

Large, polyglot service landscape.

Desire to extract service‑level governance from application code.

Common pattern: Nginx handles north‑south ingress, while Envoy/Service‑Mesh manages east‑west traffic.

Evolution Roadmap – From Single‑Node to High‑Availability System

Stage 1 – Single Node: basic reverse proxy, TLS, simple rate limiting, static serving.

Stage 2 – Dual Node HA: add Keepalived or cloud SLB for VIP, address connection drift and session stickiness.

Stage 3 – Edge Cluster: multiple Nginx instances, unified logging/metrics, automated CI/CD, layered caching.

Stage 4 – Gateway‑Level Governance: introduce config center, gray rollout, dynamic weights, plugin‑based auth/limiting.

Stage 5 – Service Mesh Integration: separate north‑south (Nginx) from east‑west (Envoy/mesh), unify observability, centralize security policies.

The key is incremental problem solving rather than a one‑shot feature dump.

Conclusion – Stability Over Speed

High‑concurrency architectures are fundamentally about traffic governance: shaping inbound flow, isolating faults, offloading via cache, and evolving configuration safely. Nginx’s true value lies in providing a rock‑solid first‑hop that protects downstream services, not merely in raw request‑per‑second throughput.