Nginx High-Performance Tuning & Reverse Proxy Best Practices: From Default to Production-Ready

1. Problem Background: Misconfiguration, Not Missing Parameters

The author presents three real production scenarios:

Scenario 1: QPS stuck, CPU low. An 8-core Nginx with four Java backends hit 3000 QPS but latency jumped from 30ms to 800ms. top showed Nginx workers at 40% CPU, ss -s revealed 80k+ TIME_WAIT sockets, while established connections to backend were only hundreds. Root cause: upstream keepalive not enabled, causing a new TCP connection per request, exhausting backend accept queues and ports.

Scenario 2: Large file downloads OOM. Static site with proxy_buffering on, proxy_buffers 8 1m, proxy_busy_buffers_size 8m. 1500 concurrent 2GB downloads consumed 12GB memory, triggering OOM Killer. The fix is not smaller buffers but disabling proxy_buffering for large files.

Scenario 3: One wrong if causes 502. An if ($http_user_agent ~* "curl") { return 403; } inside location broke proxy_pass URI handling due to Nginx's "evil if" behavior. Correct approach: use map or separate location blocks.

Common theme: none were solved by tweaking worker_processes. Tuning requires identifying the bottleneck layer first.

2. Applicable & Non-Applicable Scenarios

Applicable

Reverse proxy gateway (connection reuse, timeouts, canary, rate limiting)

Static asset / object storage frontend (sendfile, caching, Range support)

TLS termination (HTTP/2, OCSP Stapling, session reuse)

Multi-site virtual hosts (efficient server_name matching)

API gateway basics (routing, auth header pass-through, body size limits, audit logs)

Non-Applicable

Complex business logic gateways → use OpenResty/APISIX

Service mesh internal traffic → Envoy better for xDS, mTLS

Frequent dynamic config changes → open source Nginx requires reload

Pure Layer 4 load balancing → LVS/HAProxy more efficient

Million-scale long connections → need kernel/memory tuning + LVS + multiple Nginx

Version Boundaries

Based on open-source Nginx 1.30.x (1.30.6 baseline). Key differences between open-source and NGINX Plus highlighted: dynamic upstream API, active health checks, sticky, least_time, JWT/OIDC, real-time dashboard are Plus-only. Using Plus directives in open-source causes unknown directive errors.

3. Acceptance Criteria: Defining "High Performance"

Five dimensions with metrics and collection methods:

Throughput : QPS/RPS — Load test + stub_status — Target: 1.5× backend capacity headroom

Latency : P50/P95/P99 — Access log $request_time — Target: P99 < backend P99 × 1.3

Resources : Worker CPU, RSS, fd count — top, /proc/<pid>/status, ls /proc/<pid>/fd | wc -l — Target: Single worker RSS stable

Connections : active/reading/writing/waiting — stub_status — Target: waiting 60%-80% normal

Errors : 5xx ratio, upstream_* counters — Log + stub_status — Target: 5xx < 0.1%

Critical distinction: $request_time includes backend + network + client; $upstream_response_time is backend only. Difference reveals Nginx/client-side issues. Log format example provided to capture both.

4. Environment Assumptions

CentOS Stream 9 / Rocky Linux 9 (kernel 5.14+), Nginx 1.30.x from nginx.org repo, 8-core CPU, 16GB RAM, Java/Go backends, OpenSSL 3.0+, single node behind LVS/cloud SLB. Installation via official repo recommended for consistent modules (e.g., --with-http_v2_module). Archive nginx -V output for upgrade comparison.

5. Pre-Flight Checks

5.1 Version & Compile Parameters

nginx -V

must show --with-http_ssl_module, --with-http_v2_module, --with-http_realip_module, --with-http_stub_status_module. Missing modules cause silent failures (e.g., http2 on ignored).

5.2 Config Syntax & Effective State

nginx -t

checks syntax only; nginx -T dumps full effective config (including includes). Semantic errors (wrong upstream name, expired cert, missing root dir) pass syntax check.

5.3 File Descriptor Limits

Systemd drop-in /etc/systemd/system/nginx.service.d/limits.conf with LimitNOFILE=102400. Verify via cat /proc/$(cat /run/nginx.pid)/limits. Must exceed worker_processes × worker_connections (e.g., 8 × 10240 = 81920, plus 20% margin). Requires restart, not reload.

5.4 Kernel Parameters

Key sysctl settings: net.core.somaxconn=65535, net.ipv4.tcp_max_syn_backlog=65535, net.ipv4.ip_local_port_range=10240 65000, net.ipv4.tcp_tw_reuse=1, net.ipv4.tcp_fin_timeout=15, TCP buffer sizes. ip_local_port_range is hidden bottleneck for reverse proxy; expanding it only delays exhaustion without keepalive.

5.5 Backend Keepalive Support

Verify with curl -sv for Connection: keep-alive (or absence of Connection: close). Tomcat: maxKeepAliveRequests, connectionTimeout. Go: Server.IdleTimeout. PHP-FPM: fastcgi_keep_conn on. Backend keepalive_timeout must exceed Nginx's upstream keepalive_timeout to avoid upstream prematurely closed connection.

6. Core Knowledge: Why Nginx Is Fast

6.1 Process Model: Master + Workers

Master reads config, binds ports, manages workers. Workers handle all requests. Implications: worker_processes auto (equals CPU cores); each worker is single-threaded event loop — any blocking call (network FS, slow auth_request) stalls entire worker; shared state requires explicit shared memory zones ( limit_req_zone, proxy_cache_path keys_zone, upstream zone). Zone exhaustion logs zone "xxx" is exhausted.

6.2 Event Model: epoll & worker_connections

Linux uses epoll. worker_connections 10240 per worker; multi_accept on accepts all queued connections per loop; accept_mutex off (modern kernels use EPOLLEXCLUSIVE). Capacity formula: max concurrent clients = worker_processes × worker_connections / 2 (reverse proxy uses two fds per client). Required fd limit ≈ worker_processes × worker_connections × 1.2.

6.3 Connection Lifecycle & Four States

stub_status

shows: accepts / handled (mismatch = dropped connections), requests / handled ≈ 1 means keepalive broken, Reading (reading request header), Writing (writing response), Waiting (keepalive idle, should be 60-80% of active).

6.4 Memory Consumers

Connection struct (~1-2KB)

Request header buffer ( client_header_buffer_size, large_client_header_buffers)

Request body buffer ( client_body_buffer_size, client_max_body_size)

Proxy response buffers ( proxy_buffers, proxy_buffering)

Temp files ( proxy_temp_path, proxy_max_temp_file_size)

Cache ( proxy_cache_path max_size)

Shared memory zones (allocated at start)

TLS session cache ( ssl_session_cache shared:SSL:10m ≈ 40k sessions)

Biggest risk: proxy_buffering on with large responses — default 8×4k/8k buffers per connection; increasing to 8×1m makes 1000 concurrent downloads consume 24GB.

7. Implementation Roadmap

Six sequential phases: ① System tuning (kernel, fd, FS) → ② Process & event model → ③ Static file path (sendfile, open_file_cache) → ④ Reverse proxy core (keepalive, buffers, timeouts, retries) → ⑤ Protective config (rate limiting, limits, cache) → ⑥ TLS/HTTP2/3 optimization. Each step must verify: config effective ( nginx -T | grep), behavior changed (metrics), no regressions (e.g., new upstream prematurely closed). Reload safety: worker_processes / worker_connections changes double memory/fds during reload; listen port changes need firewall rules first; cert changes need full chain validation; limit rules should start with dry_run.

8. System-Level Tuning

File descriptors via systemd drop-in (requires restart). Kernel params via /etc/sysctl.d/99-nginx.conf with rationale for each. Log optimization: access_log ... buffer=64k flush=5s; conditional logging via map to skip static assets; logrotate with postrotate kill -USR1 to reopen files.

9. Static File Serving

sendfile on

avoids user-space copy. tcp_nopush on + tcp_nodelay on work together: headers sent immediately, body coalesced.

open_file_cache max=10000 inactive=60s valid=80s min_uses=2 errors=on

caches file metadata. Complete location /static/ config with alias, expires 30d, Cache-Control: public, immutable, aio threads, directio 4m, gzip_static on. max_ranges 1 prevents Range amplification attacks. alias vs root distinction critical.

10. Reverse Proxy Connection Reuse (Highest ROI)

Version Changes (1.29.7+)

proxy_http_version

default changed from 1.0 to 1.1; upstream keepalive enabled by default (32 per worker); Connection header no longer sent upstream by default. Still recommend explicit config for reliability.

Correct Upstream Config

upstream app_backend { zone app_backend 64k; least_conn; server 10.0.1.11:8080 max_fails=3 fail_timeout=10s; server 10.0.1.12:8080 max_fails=3 fail_timeout=10s; server 10.0.1.13:8080 max_fails=3 fail_timeout=10s; keepalive 64; keepalive_requests 1000; keepalive_timeout 60s; keepalive_time 1h; }

keepalive 64

per worker (total 512 idle connections). keepalive_requests 1000 forces periodic rebalancing (prevents new instances getting zero traffic). keepalive_timeout must be less than backend's idle timeout. keepalive_time 1h max connection lifetime. New keepalive 64 local (1.29.7+) reduces cross-worker lock contention.

Proxy Block Config

Key directives: proxy_http_version 1.1, proxy_set_header Host $http_host (vs $host which falls back to server_name), proxy_set_header Connection "" (clears client's Connection header), proxy_buffering on with proxy_buffers 8 8k for normal APIs; off for SSE, large downloads, WebSocket. Timeouts: proxy_connect_timeout 3s, proxy_send_timeout 30s, proxy_read_timeout 30s (interval between reads, not total time). Retries: proxy_next_upstream error timeout only for idempotent methods (GET/HEAD) via map $request_method $can_retry.

Verification

Check ss -tn state established '( dport = :8080 )' stable; 100 requests should generate single-digit SYN packets;

stub_status

requests/handled

> 10. Real benchmark: adding keepalive boosted QPS from 2800 to 14500, P99 from 780ms to 42ms, eliminated TIME_WAIT exhaustion.

11. Timeout System & Failure Isolation

Three-layer timeout hierarchy: client (60s) > Nginx (30s) > backend (20s). Full timeout list: client header/body/send/keepalive; proxy connect/send/read; proxy_socket_keepalive on for dead peer detection (supplementary). Slow client mitigation: client_header_timeout 10s, client_body_timeout 20s, limit_conn_zone with limit_conn_dry_run on for observation. Passive health checks via max_fails=3 fail_timeout=10s with zone for shared counters. Graceful drain: mark server down and reload. 502/504/499 diagnosis table with error.log patterns and actionable checks.

12. Rate Limiting & Anti-Avalanche

limit_req_zone $binary_remote_addr zone=api_perip:10m rate=20r/s

; limit_req zone=api_perip burst=40 nodelay (burst allows spikes, nodelay processes burst immediately). Multi-level: per-IP then per-vhost. Path-specific limits via map returning empty key to skip. Global limit via $server_name. Tenant-based limit via header (requires trusted header). Verification with wrk and log analysis. Open-source lacks true circuit breaker; approximations: aggressive max_fails, error_page fallback with default_type application/json before return.

13. Proxy Cache Strategy

proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=api_cache:100m max_size=10g inactive=60m use_temp_path=off

. use_temp_path=off avoids cross-device copy. Location config: proxy_cache_valid 200 301 302 10m, 404 1m,

proxy_cache_use_stale error timeout updating http_500 502 503 504

, proxy_cache_background_update on, proxy_cache_lock on with 5s timeout. Cache key design: include $http_accept_encoding, language via map; avoid cookies, remote_addr. $upstream_cache_status values: MISS, HIT, EXPIRED, STALE (alert), UPDATING, REVALIDATED, BYPASS. Cache purge: delete files + reload (causes thundering herd), third-party ngx_cache_purge, or versioned URLs (preferred).

14. TLS & HTTP/2/3

Modern TLS: ssl_protocols TLSv1.2 TLSv1.3, AEAD ciphers, ssl_prefer_server_ciphers off, ssl_session_cache shared:SSL:50m, ssl_session_tickets on, OCSP Stapling (declining value with Let's Encrypt). HTTP/2: http2 on (1.25.1+), http2_max_concurrent_streams 128, no server push (removed), use early_hints. HTTP/3: listen 443 quic reuseport, add_header Alt-Svc 'h3=":443"; ma=86400', ssl_early_data on only for idempotent GET. TLS overhead optimization order: session reuse > ECDSA certs > TLS 1.3 > hardware offload. Log $ssl_protocol, $ssl_cipher, $ssl_session_reused to monitor reuse rate (target >60%).

15. Logging & Observability

Minimal log format (10 fields) plus tracing format (+5 fields). Key variables: $request_time (total), $upstream_response_time (comma-separated for retries), $upstream_addr, $upstream_status, $request_id (must propagate to backend via X-Request-ID). awk one-liners for top URLs, latency percentiles, status codes, 5xx URLs, per-minute QPS, upstream time sums. Prometheus via nginx-prometheus-exporter (only 7 stub_status metrics) or log-to-metric tools ( mtail, grok_exporter). Alert thresholds: 5xx >1%, P99 >2× baseline, accepts-handled growing, waiting/active <20%, 499 >5%, STALE cache status, upstream connect failures.

16. Troubleshooting Six Scenarios

QPS stuck, CPU low: Check requests/handled ≈1, TIME_WAIT high, established connections to backend low, TcpExtListenOverflows. Fix: enable keepalive.

Intermittent 502: upstream prematurely closed connection → backend keepalive shorter than Nginx. Fix: Nginx keepalive_requests 90 < backend's 100, keepalive_timeout 50s < backend idle timeout.

504 spikes: urt near proxy_read_timeout. Check backend thread pool, DB, downstream. Mitigate: increase timeout temporarily, rate limit slow endpoint, fix root cause.

Large upload fails: 413 → client_max_body_size; 502 during upload → client_body_temp_path disk full. Use proxy_request_buffering off for streaming (conflicts with retries).

Reload not effective: Check nginx -T for actual config, include files, multiple nginx.conf, server_name match, location priority (= > ^~ > regex > prefix).

Memory growth: RSS rising → check proxy_buffering with large responses, cache zone size, connection leaks, third-party modules. Pss vs Rss distinguishes shared memory. Periodic reload mitigates but not root cause.

17. Command Cheatsheet

Categorized commands for config validation ( nginx -t/T), connections ( ss, stub_status), processes ( ps, fd counts), logs ( awk analysis), load testing ( wrk with -k for keepalive), TLS ( openssl s_client), packet capture ( tcpdump, nstat, perf). Key nstat metrics: TcpExtListenOverflows, TcpExtListenDrops, TcpExtTCPAbortOnClose.

18. Production Discipline

Config as code: Git, PR review, staged rollout (canary single node). Reload boundaries: long connections delay worker exit (memory double), cache cold start. Change windows: low-traffic for upstream/worker changes, anytime for static rules. Upgrade checklist: compare nginx -V modules, check deprecations, read release notes "Change" section, full regression, rollback package ready. Security: server_tokens off, restrict stub_status, block hidden files, prevent script execution in uploads. Track CVEs via official advisories.

19. Complete Production Config

Two-file example: nginx.conf (main, events, http, logging, base limits, cache path, upstream, status server) and conf.d/api.conf (HTTPS server with HTTP/2, TLS, rate limits, API locations with/without cache, SSE, static assets, fallback 404). Annotations explain worker_rlimit_nofile, per-location open_file_cache, dual-level rate limiting.

20. Summary

Four actionable principles: (1) Measure first, tune later — establish baseline with stub_status, $request_time, ss -s. (2) Bottlenecks live in connection layer, not parameter layer — keepalive, fd limits, backend keepalive mismatch are typical culprits. (3) Defaults change; write critical directives explicitly ( proxy_http_version, Connection, keepalive series). (4) Protective config (rate limits, timeouts, max_fails, proxy_cache_use_stale) matters more than performance config — they prevent avalanches. Three long-term mechanisms: config change regression checklist, stable log field contract ( $request_id, rt, urt, ustat), version upgrade default-value diff log.