Nginx Optimization Guide: Doubling Concurrency via Configuration Tuning

Background

A typical scenario: a 4-core 8GB machine running Nginx reverse proxy to 4 PHP-FPM servers. Under default configuration ( worker_processes 1, worker_connections 1024, proxy_buffers 8 4k, no gzip, no keepalive, no rate limiting), concurrency capped at 2000 with 5xx errors. After systematic tuning — worker_processes auto, adjusted keepalive, buffers, gzip, client_body_buffer_size, proxy_*_timeout — wrk tests showed concurrency rising from 2000 to ~6000 on the same hardware, eliminating 502 errors under high load.

Applicable Scenarios

Short-connection QPS stalls at 1k–2k on 4-core.

Low user CPU but high kernel CPU (socket/buffer/accept queue backpressure).

wrk curves show RT jitter and 5xx at high concurrency.

High Waiting connections (slow clients consuming workers).

Backend (PHP-FPM/Java/Go/Python) RT high but bottleneck may be Nginx buffering, disk writes, or timeouts.

Bandwidth pressure: missing gzip, sendfile, tcp_nopush.

Static assets saturated: no dynamic/static separation, no expires.

Under attack: no limit_req_zone, no UA blocking.

Unsure about worker count or memory adequacy.

Out of scope: HTTPS deep tuning (session resumption, TLS 1.3, OCSP stapling), Lua/WAF logic, dynamic modules, CDN/SLB/WAF integration.

Core Knowledge Points (12)

1. Master/Worker Model

Master reads config, forks workers, handles graceful upgrades. worker_processes controls worker count: set to physical CPU cores or auto (since 1.9.10). Too many workers increase context switching.

2. worker_connections

Max simultaneous connections per worker. Theoretical max concurrency = worker_processes × worker_connections / 2 (client + upstream connection). Recommended worker_connections 4096. accept_mutex on is legacy; modern Linux defaults to off.

3. sendfile / tcp_nopush / tcp_nodelay

sendfile on

: kernel copies data between kernel buffers, avoiding user↔kernel copies. tcp_nopush on: aggregates response header and body (Nagle-like). tcp_nodelay on: disables Nagle for keepalive connections.

Enable all three in http block.

4. keepalive_timeout / keepalive_requests

keepalive_timeout 65

: client keep-alive timeout; too high consumes FDs, too low reduces reuse. keepalive_requests 1000: max requests per connection; prevents single connection monopolizing worker.

5. open_file_cache

open_file_cache max=1000 inactive=20s

caches file descriptors, size, mtime. Highly beneficial for static assets, less so for reverse proxy.

6. proxy_buffer & proxy_buffer_size

proxy_buffer_size 16k

: first buffer for response headers. proxy_buffers 8 16k: 8 buffers of 16k for response body. proxy_busy_buffers_size 32k: total size of buffers marked busy.

For large file proxying, use proxy_buffering off to avoid disk write fallback; watch for I/O jitter.

7. proxy_*_timeout

proxy_connect_timeout

: time to connect to upstream. proxy_send_timeout: time to send request to upstream. proxy_read_timeout: time to read response from upstream.

Unset timeouts can hang workers. Common values: 3s, 30s, 30s.

8. gzip

gzip on; gzip_min_length 1k; gzip_comp_level 5; gzip_types text/plain text/css application/json application/javascript ...; gzip_vary on; gzip_proxied any

. Warning: CPU cost; avoid compressing already-compressed types like application/octet-stream or image/png.

9. limit_req / limit_conn

limit_req_zone $binary_remote_addr zone=req_zone:10m rate=30r/s; limit_req zone=req_zone burst=60 nodelay;

. burst is queue limit; nodelay processes queued requests immediately (returns 503/429 when full). limit_conn_zone similarly limits concurrent connections.

10. upstream keepalive

upstream backend { server 10.0.0.7:8080; server 10.0.0.8:8080; keepalive 64; }

with proxy_http_version 1.1; proxy_set_header Connection "";. Without upstream keepalive, each request incurs new TCP/TLS handshake to backend.

11. Location Matching Priority

= /uri

exact match (highest). ^~ /uri/ prefix match, stops regex. ~ /regex case-sensitive regex. ~* /regex case-insensitive regex.

Plain prefix: longest match first, then regex.

Correct routing: exact > regex > plain prefix.

12. $request_time vs $upstream_response_time

$request_time

: total request time inside Nginx. $upstream_response_time: upstream response time. Comparing them isolates frontend vs backend slowness.

Overall Troubleshooting Approach

Optimization = evidence chain → hypothesis → experiment → verification, not random toggle flipping.

Baseline : capture current QPS, RT, 5xx rate; run wrk / ab at fixed concurrency/duration; record P50/P90/P99; collect top, iostat, ss -s.

Target optimization points (easy to hard) : worker/events/fd → keepalive/sendfile/tcp_nopush/tcp_nodelay → open_file_cache/expires → gzip → proxy_buffer/timeout → upstream keepalive → limit_req/blacklist.

Config changes : backup ( cp nginx.conf nginx.conf.bak$(date +%s)), change one item, verify with nginx -t && nginx -s reload.

Observe & compare : rerun wrk under same conditions; monitor CPU/mem/iowait/socket.

Record rollback points for each step.

Practical Steps (15)

Step 1: Baseline Load Test

Command: wrk -t8 -c200 -d30s --latency http://127.0.0.1:8080/. Record QPS, latency percentiles, errors. Also ab -n 50000 -c 200 for POST with body. Note anomalies: 99th percentile >500ms, socket timeouts.

Step 2: Baseline Machine Metrics

top -b -n 1 > top.before.txt; iostat -xz 1 10 > iostat.before.txt; vmstat 1 10 > vmstat.before.txt; ss -s > ss.before.txt

. Expect user CPU <50%, swap 0, low await, reasonable TIME_WAIT.

Step 3: Adjust Worker Count

worker_processes auto; worker_rlimit_nofile 65535;

Verify with grep -c '^processor' /proc/cpuinfo. Reload and check worker count and CPU balance.

Step 4: Events Block

events { worker_connections 4096; multi_accept on; use epoll; }

. multi_accept accepts multiple connections per event loop; use epoll explicit for Linux.

Step 5: HTTP Basic I/O

Enable sendfile, tcp_nopush, tcp_nodelay, keepalive_timeout 65, keepalive_requests 1000, types_hash_max_size 2048, server_tokens off. File cache:

open_file_cache max=10000 inactive=30s; open_file_cache_valid 60s; open_file_cache_min_uses 2; open_file_cache_errors on;

.

Step 6: Enable gzip

gzip on; gzip_min_length 1k; gzip_comp_level 5; gzip_types text/plain text/css application/json application/javascript application/xml image/svg+xml; gzip_proxied any; gzip_vary on;

(avoid application/octet-stream, image/png).

Step 7: Client Settings

client_max_body_size 16m; client_body_buffer_size 128k; client_header_buffer_size 1k; large_client_header_buffers 4 8k; client_body_timeout 30s; client_header_timeout 30s; send_timeout 30s; reset_timedout_connection on;

.

Step 8: Reverse Proxy Key Params

HTTP block:

proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_connect_timeout 3s; proxy_send_timeout 30s; proxy_read_timeout 30s; proxy_buffer_size 16k; proxy_buffers 8 16k; proxy_busy_buffers_size 32k; proxy_temp_file_write_size 8k; proxy_buffering on;

. Upstream block with keepalive 64 and max_fails=3 fail_timeout=10s. Location uses proxy_pass http://backend;.

Step 9: Static/Dynamic Separation

Location /static/ with alias, expires 7d, Cache-Control: public, max-age=604800, immutable, access_log off. Regex location for extensions with same headers and try_files $uri =404.

Step 10: FastCGI (PHP)

fastcgi_pass unix:/run/php/php-fpm.sock; fastcgi_buffer_size 16k; fastcgi_buffers 8 16k; fastcgi_busy_buffers_size 32k; fastcgi_read_timeout 60s;

(align with PHP-FPM timeout).

Step 11: Rate Limiting, Anti-hotlink, HTTPS

limit_req_zone $binary_remote_addr zone=req_zone:10m rate=30r/s; limit_conn_zone $binary_remote_addr zone=conn_zone:10m;

. SSL:

ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:...; ssl_session_cache shared:SSL:10m; ssl_session_timeout 1d; ssl_session_tickets on;

.

Step 12: Log Format

JSON log format with $request_time, $upstream_response_time, $upstream_status, etc.

access_log /var/log/nginx/access.log main buffer=32k flush=5s;

.

Step 13: Reload vs Restart

Use nginx -t && nginx -s reload for config changes. Restart only for *_temp_path changes. Binary upgrade: kill -USR2 + kill -QUIT.

Step 14: Re-test & Compare

Run same wrk command, save to wrk.after.txt, compare QPS/P99 with awk 'NR==13 || NR==27' wrk.before.txt wrk.after.txt.

Step 15: Stability Observation

tail -F /var/log/nginx/access.log | jq -r '.request_time' | awk '{s+=$1} END {print "avg:", s/NR}'

. Watch 5xx rate, avg/P99 RT.

Common Commands Reference

Tables for Nginx control ( nginx -t, nginx -T, nginx -s reload, etc.), process inspection ( ps, /proc), load testing ( wrk, ab, hey), performance monitoring ( top, pidstat, perf, iostat), log analysis ( awk, jq), status ( stub_status, curl -w).

Complete Configuration Examples

Provided reusable nginx.conf, site.conf (upstream, server, SSL, location, proxy, static), proxy_params, gzip.conf, logrotate, and PHP-FPM www.conf (pm.max_children=50 based on 30-100MB per child).

Log & Metric Observation Methods

1. CPU

top -b -n 1 | head -5; mpstat -P ALL 1 5

. Check per-core balance, us+sy reduction.

2. Memory

free -h; ps -eo pid,pmem,rss,comm | grep nginx

. Worker RSS typically 100-300MB.

3. Disk I/O

iostat -xz 1 5

. Target await <5ms for proxy; %util low; watch w/s for access log writes.

4. Network

ss -s; sar -n DEV 1 5

. ESTABLISHED up, TIME_WAIT down after keepalive.

5. Nginx Status

curl -s http://127.0.0.1:8080/nginx_status

. Monitor Active, Reading, Writing, Waiting. High Waiting indicates keepalive pile-up.

6. Alert Thresholds

5xx >1%, P99 RT >1s (warn) >3s (critical), Active connections >80% capacity, RSS >30% RAM, disk await >30ms.

7. Log Analysis

awk '{if ($9 ~ /^5/) print $7, $9}' access.log | sort | uniq -c | sort -nr | head

for top 5xx URLs.

8. Request Time vs Upstream Time

Awk script to extract requests where $request_time > 0.5 and compare with $upstream_response_time to pinpoint frontend vs backend delay.

Troubleshooting Paths (15 Decision Trees)

Each path: symptom → checks → remediation. Examples:

Slower after tuning : check keepalive_requests too low, proxy_buffering on with slow disk, gzip CPU saturation.

5xx spike : increase proxy_read_timeout, proxy_connect_timeout, upstream keepalive, relax max_fails.

Too many open files : raise worker_rlimit_nofile, ulimit -n, /proc/sys/fs/file-max.

Memory growth : reduce proxy_buffers total size, gzip_buffers.

High disk I/O : disable proxy_buffering for large responses, increase access_log buffer.

Client 504 : investigate backend (PHP-FPM, DB slow query).

$request_time high, $upstream_response_time low : client upload slow, gzip CPU, log write, network.

Single CPU core 100% : uneven upstream keepalive, CPU-heavy Lua/regex/JSON.

False rate limiting : adjust rate, add internal IP whitelist, verify real IP header.

HTTPS slower : enable ssl_session_cache, ssl_session_tickets, TLS 1.3, modern ciphers.

Config not effective : verify with nginx -T, stub_status, wrk.

Log loss : access_log buffer=32k flush=5s may lose 5s of logs; access_log off in location.

Cache hit rate : verify Cache-Control, ETag, Last-Modified via curl -I.

Worker unresponsive after reload : old workers finish requests; never kill -9.

Cross-origin/auth latency : auth_request subrequests, sync third-party calls; async via Redis queue.

Risk Warnings (17)

Key risks: changing everything at once (no causality), workers > cores×8 (context switch), worker_connections 65535 (fd exhaustion), keepalive_requests 1000000 (resource hog), gzip on without gzip_types (only HTML compressed), gzip_comp_level 9 (CPU spike), proxy_buffering off for large responses (disk thrash), excessive proxy_read_timeout (worker hang), sendfile with directio conflict, too strict limit_req (false 429), duplicate add_header + expires (browser confusion), upstream keepalive hotspots (use least_conn), weak SSL ciphers, log_format syntax errors, wrong proxy_set_header Host, small large_client_header_buffers (400 errors), large proxy_temp_file_write_size (I/O jitter).

Verification Checklist

nginx -t

→ syntax ok. nginx -s reload → master PID unchanged, worker PIDs rotated. curl -s http://127.0.0.1:8080/nginx_status → local only.

Static: Cache-Control: public, max-age=604800, immutable; dynamic: none. curl -H 'Accept-Encoding: gzip' -I → Content-Encoding: gzip.

Rate limit: loop curl → mix of 200 and 429.

Real IP: curl -H 'X-Forwarded-For: 1.2.3.4' → backend sees 1.2.3.4.

Client cache: second request with If-Modified-Since → 304.

Load test before/after → QPS up, P99 down. dmesg | tail; journalctl -p err --since "15 min ago" → no new errors.

Canary rollout 1%→5%→50%→100%; no new alerts.

Rollback Plan

Backup:

cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak$(date +%Y%m%d-%H%M%S)

and conf.d.

After change: nginx -t && nginx -s reload.

Rollback: restore backup files, nginx -t && nginx -s reload.

SSL cert changes: test on alternate port (8443) first.

Upstream changes: keep old IP list; revert on 502.

Log format: keep original format string.

Emergency: nginx -s stop; cp backup; nginx.

Container: keep previous image for instant rollback.

Production Notes

Test in staging mirroring prod (CPU, RAM, concurrency).

Always nginx -s reload, never restart unless worker deadlock.

Worker count: lscpu for cores; in containers, limit below host cores.

Large files: aio threads (1.7+), sendfile + directio.

Health checks: max_fails=3 fail_timeout=10s, proxy_next_upstream.

Upstream keepalive: start at 64; too high may exhaust backend resources.

Slow client timeouts: client_body_timeout, client_header_timeout, send_timeout.

Avoid log_nostream on (explodes I/O).

Health endpoint: use /healthz not stub_status.

Include order matters: include /etc/nginx/conf.d/*.conf loads lexicographically; map must be in http block top.

Upgrade: keep compile flags via nginx -V.

CDN: set real_ip_header + set_real_ip_from; for third-party proxy, proxy_ssl_server_name on.

TOCTOU: access_log off in server/location, not http; avoid variable mutation in if.

LB algorithm: prefer least_conn.

Proxy cache: separate buckets per URL pattern.

Containers: setcap cap_net_bind_service=+ep for non-root port 80; config via ConfigMap.

Dashboard metrics: Active connections, Req/s, P99 RT, 5xx%, geo distribution, upstream RT.

Emergency cheat sheet: nginx -t, nginx -T | grep, tail -F error.log, ss -tunlp, wrk, dmesg, journalctl -u nginx, curl -v.

Summary

Nginx optimization boils to four pillars:

Foundation : worker, events, fd limits, keepalive.

I/O & Bandwidth : sendfile, tcp_nopush, tcp_nodelay, gzip, open_file_cache.

Efficient Proxy : proper buffers, timeouts, upstream keepalive, real_ip.

Overload Protection : limit_req, UA blacklist, connection limits, status codes.

Each knob has clear "why enable" and "what if misconfigured". Core principle: tune per workload + validate with monitoring and load tests. The workflow — goal → environment assumptions → baseline → incremental change → evidence chain — must be scripted, canaried, and rollback-ready. Next step: codify as IaC (Ansible + CI wrk regression) so every optimization becomes reproducible code.