Which Nginx Parameters Should You Tune First for High‑Concurrency Scenarios?

Background and Problem

Nginx defaults are tuned for low‑traffic workloads; when QPS reaches thousands or tens of thousands, the server shows high CPU idle time, request queuing, and connection refusals.

Parameter Adjustment Overview and Priority

Adjustments follow a strict priority order: OS kernel limits → Nginx process model → per‑connection buffers → upstream & keep‑alive → HTTP‑level optimizations. Each layer should be tuned, verified, then the next layer applied.

1. Kernel Parameters

1.1 File‑Descriptor Limits

Each connection consumes a file descriptor (FD). The default 1024 is far too low.

# View system‑wide max FD
cat /proc/sys/fs/file-max
# Current FD usage
cat /proc/sys/fs/file-nr
# Nginx soft limit
cat /proc/$(cat /var/run/nginx.pid)/limits | grep "Max open files"

Increase the limits:

# /etc/sysctl.conf
fs.file-max = 1000000
# Apply
sysctl -p

# /etc/security/limits.conf
nginx soft nofile 100000
nginx hard nofile 100000

worker_rlimit_nofile 100000;

The formula for theoretical max connections is worker_rlimit_nofile / 2 because each connection needs an FD plus extra for logs, upstream, etc.

1.2 Network Kernel Settings

# /etc/sysctl.conf additions
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15000
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.ip_local_port_range = 1024 65535
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

Apply with sysctl -p. The parameters control the listen backlog, SYN queue, TIME_WAIT reuse, keep‑alive intervals, and socket buffer sizes.

2. Nginx Process Model

2.1 Worker Processes

Use a multi‑process model; the default single worker cannot utilize multi‑core CPUs.

# Detect CPU cores
nproc
# Or
grep processor /proc/cpuinfo | wc -l

worker_processes auto;

"auto" sets the number of workers to the CPU core count. Over‑provisioning adds context‑switch overhead.

2.2 Worker Connections

events {
    worker_connections 65535;
    use epoll;
    multi_accept on;
}

use epoll

selects the most efficient I/O multiplexing on Linux. multi_accept on lets a worker accept multiple connections per loop, reducing queuing.

The total possible connections are worker_connections * worker_processes and must stay below worker_rlimit_nofile.

2.3 CPU Affinity

worker_cpu_affinity auto;

"auto" binds each worker to a distinct core. Manual masks (e.g., worker_cpu_affinity 0001 0010 0100 1000;) are useful on NUMA servers.

2.4 Worker Priority

worker_priority -10;

Negative nice values give Nginx higher CPU priority; adjust only when other critical services share the host.

3. Buffer Configuration

3.1 Client Buffers

http {
    client_header_buffer_size 4k;
    large_client_header_buffers 4 32k;
    client_body_buffer_size 128k;
    client_max_body_size 100m;
}

Small headers fit in 4 KB; larger ones use the 4×32 KB pool. Body buffers larger than 128 KB avoid frequent disk writes.

3.2 Upstream Buffers

upstream backend {
    server 127.0.0.1:8080;
    keepalive 32;
}
proxy_buffer_size 128k;
proxy_buffers 4 128k;
proxy_buffering on;
proxy_busy_buffers_size 256k;

These settings let Nginx asynchronously read upstream responses, reducing worker blocking.

3.3 FastCGI Buffers (PHP‑FPM)

fastcgi_buffer_size 64k;
fastcgi_buffers 4 64k;
fastcgi_busy_buffers_size 128k;
fastcgi_temp_file_write_size 256k;
fastcgi_connect_timeout 60s;
fastcgi_send_timeout 60s;
fastcgi_read_timeout 60s;

Adjust timeouts for long‑running PHP scripts.

4. Timeout Settings

http {
    keepalive_timeout 65;
    client_header_timeout 15s;
    client_body_timeout 15s;
    send_timeout 30s;
}

Balance keep‑alive duration, client request timeouts, and server response timeout to avoid idle connections consuming resources.

4.1 Upstream Keep‑Alive

upstream backend {
    server 127.0.0.1:8080;
    keepalive 64;
    keepalive_requests 10000;
    keepalive_timeout 60s;
}

Each worker maintains a pool of idle connections; after 10 000 requests a connection is recycled to prevent leaks.

5. Compression and Transfer Optimisation

5.1 Gzip

http {
    gzip on;
    gzip_vary on;
    gzip_min_length 1024;
    gzip_proxied any;
    gzip_comp_level 4;
    gzip_types text/plain text/css text/xml application/json application/javascript application/xml application/xml+rss;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_disable "MSIE [1-6]\.";
}

Level 4 offers a good CPU‑to‑compression ratio; small responses (<1 KB) are left uncompressed.

5.2 Static Resource Caching

location ~* \.(jpg|jpeg|png|gif|ico|css|js|woff|woff2|ttf|eot)$ {
    expires 30d;
    add_header Cache-Control "public, no-transform";
    access_log off;
}

Long‑term caching reduces repeat requests for immutable assets.

5.3 Sendfile Zero‑Copy

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
}

Eliminates user‑space copies when serving static files.

6. Connection Handling Optimisation

6.1 HTTP/2

http {
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256';
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;
}
server {
    listen 443 ssl http2;
    server_name example.com;
}

Enables multiplexing, header compression, and server push.

6.2 Rate Limiting and Connection Limiting

limit_req_zone $binary_remote_addr zone=req_limit:10m rate=1000r/s;
limit_conn_zone $binary_remote_addr zone=conn_limit:10m;
server {
    location / {
        limit_req zone=req_limit burst=2000 nodelay;
        limit_conn conn_limit 100;
        proxy_pass http://backend;
    }
}

Controls request bursts and per‑IP concurrent connections.

7. Caching Configuration

proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=api_cache:100m max_size=10g inactive=60m use_temp_path=off;
server {
    location /api/ {
        proxy_pass http://backend;
        proxy_cache api_cache;
        proxy_cache_valid 200 60s;
        proxy_cache_valid 404 10s;
        proxy_cache_use_stale error timeout updating;
        add_header X-Cache-Status $upstream_cache_status;
    }
}

Cache key zone of 100 MB holds ~1 M keys; stale responses are served on upstream errors.

8. Load‑Balancing Strategy Selection

8.1 Round‑Robin & Weighted

upstream backend {
    server 127.0.0.1:8080 weight=5;
    server 127.0.0.1:8081 weight=3;
    server 127.0.0.1:8082 weight=2;
}

8.2 Least Connections

upstream backend {
    least_conn;
    server 127.0.0.1:8080;
    server 127.0.0.1:8081;
}

8.3 IP Hash

upstream backend {
    ip_hash;
    server 127.0.0.1:8080;
    server 127.0.0.1:8081;
    server 127.0.0.1:8082;
}

Ensures session affinity.

9. High Availability & Disaster Recovery

9.1 Passive Health Checks

upstream backend {
    server 127.0.0.1:8080 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:8081 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:8082 max_fails=3 fail_timeout=30s;
}

Servers are marked down after three consecutive failures for 30 s.

9.2 Backup & Down

upstream backend {
    server 127.0.0.1:8080 weight=5;
    server 127.0.0.1:8081 weight=3;
    server 127.0.0.1:8082 backup;  # used only when others fail
}

9.3 Active‑Passive with Keepalived

# Keepalived VRRP configuration (excerpt)
virtual_server 192.168.1.100 443 {
    delay_loop 6
    lb_algo rr
    lb_kind VRRP
    protocol TCP
    real_server 192.168.1.101 443 { weight 1; TCP_CHECK { connect_timeout 3; nb_get_retry 3; delay_before_retry 3; } }
    real_server 192.168.1.102 443 { weight 1; TCP_CHECK { connect_timeout 3; nb_get_retry 3; delay_before_retry 3; } }
}

VIP fails over automatically when the master Nginx becomes unavailable.

10. Production Validation Checklist

Verify Nginx syntax: nginx -t Confirm file‑descriptor limits: ulimit -n shows ≥ 100000

Check worker count: ps aux | grep nginx shows multiple workers

Ensure ports are listening: ss -tlnp | grep nginx Validate upstream connection pool size via netstat -an Test rate limiting with ab or wrk Confirm cache hits via X-Cache-Status header

Compare QPS and P99 latency before and after tuning

11. Common Pitfalls

11.1 Worker Processes

Setting worker_processes higher than CPU cores adds context‑switch overhead without benefit.

11.2 Worker Connections

Do not set worker_connections equal to the system FD limit; keep it at 60‑70 % of worker_rlimit_nofile to leave room for logs, upstream sockets, etc.

11.3 Gzip Level

Level 4 (or 5) balances compression ratio and CPU cost; higher levels give diminishing returns.

11.4 Proxy Buffering

Disable proxy_buffering for real‑time streams (e.g., SSE) where latency matters.

12. Conclusion

Effective Nginx tuning for high concurrency follows the hierarchy: kernel parameters → process model → buffers → protocol features. Adjust each layer, observe metrics, and iterate. Regular reviews are essential because traffic patterns, upstream performance, and business logic evolve, potentially invalidating previous optimal settings.