Boost Nginx Performance: Scaling from Thousands to Ten‑Thousands RPS

Performance Baseline: Define “Fast”

Before changing worker_connections, write down the request model and acceptance criteria:

Identify whether traffic is static files, reverse proxy, upload, download, SSE, WebSocket, or mixed.

Determine if HTTPS, HTTP/2, keepalive are used; note client connection count, requests per connection, and new‑connection rate.

Record request/response sizes, cache hit rate, upstream protocol and normal response time.

Acceptance metrics include success rate, RPS, P50/P95/P99 latency, connection errors, CPU, memory, file descriptors, NIC, disk and upstream latency.

Ensure the load‑testing client has enough instances, bandwidth, CPU and temporary ports to avoid it becoming the bottleneck.

Do not treat the highest observed RPS as the conclusion; always collect status codes, timeouts and tail latency. A test that returns many 502/504 errors has no business value.

Environment Check Before Implementation

Verify actual Nginx version, compiled modules, run mode and effective configuration:

nginx -v
nginx -V
sudo nginx -T
systemctl status nginx --no-pager
systemctl show nginx -p LimitNOFILE -p ExecStart
ps -eo pid,ppid,cmd | rg '[n]ginx'

Example output:

nginx version: nginx/1.24.0
configure arguments: --with-http_ssl_module --with-http_v2_module

nginx -T

shows the full effective config, which may contain internal domains, certificate paths and upstream addresses that need to be sanitized.

Check worker, file‑descriptor and listening status:

sudo nginx -t
getconf _NPROCESSORS_ONLN
cat /proc/$(pgrep -o nginx)/limits | rg 'Max open files'
ss -ltnp | rg ':80|:443'
ss -s

The shell’s ulimit -n differs from the limit seen by the systemd‑started Nginx; use /proc/<nginx master pid>/limits and systemctl show as the source of truth.

Establish a Comparable Baseline

If stub_status is enabled, restrict access to trusted networks. Example loopback‑only server:

server {
    listen 127.0.0.1:8080;
    server_name _;
    location = /nginx_status {
        stub_status;
        allow 127.0.0.1;
        deny all;
    }
}

Collect baseline metrics with:

curl -sS http://127.0.0.1:8080/nginx_status
ss -s
pidstat -dur -p "$(pgrep -d, -f 'nginx: worker process')" 1 10
mpstat -P ALL 1 10
free -h
df -h

Example output (illustrative only):

Active connections: 120
server accepts handled requests
 3000 3000 5500
Reading: 1 Writing: 20 Waiting: 99

Use a consistent load‑testing tool, e.g. wrk:

wrk -t4 -c200 -d30s --latency https://example.internal/healthz

Validate status codes and response bodies; separate TLS handshake cost from keep‑alive reuse when testing HTTPS.

Bottleneck Identification: From Symptom to Evidence

1. Worker CPU, Soft IRQs and CPU Quota

pidstat -t -p "$(pgrep -d, -f 'nginx: worker process')" 1 10
mpstat -P ALL 1 10
cat /proc/softirqs
cat /proc/interrupts

High worker CPU may stem from TLS, gzip, complex rewrites, WAF, log I/O or upstream processing. Only consider increasing worker_processes or setting CPU affinity when a single core is saturated while others are idle; do not assume worker_processes auto; is optimal for every cgroup.

2. File Descriptors and Connection Limits

cat /proc/$(pgrep -o nginx)/limits | rg 'Max open files'
ls /proc/$(pgrep -o nginx)/fd | wc -l
ss -tan state time-wait | wc -l
grep -R 'too many open files' /var/log/nginx 2>/dev/null

In reverse‑proxy scenarios, the total possible client connections is not simply worker_processes * worker_connections. Evaluate worker_connections, worker_rlimit_nofile, systemd LimitNOFILE, kernel fs.file-max and upstream fd limits together. Adjust limits only when logs show “too many open files” and connection failures coincide.

3. Distinguishing Upstream Slowness from Nginx Slowness

log_format upstream_timing '$remote_addr $request $status '
    'request_time=$request_time '
    'upstream_connect_time=$upstream_connect_time '
    'upstream_header_time=$upstream_header_time '
    'upstream_response_time=$upstream_response_time';

If $upstream_*_time contains comma‑separated values (multiple retries), treat it as a list, not a single number. When both $request_time and $upstream_response_time rise, investigate upstream queues, databases, CPU, GC and connection pools; adding more Nginx workers will not speed up a slow upstream.

4. Static Files, Disk and Log I/O

iostat -xz 1 10
pidstat -d -p "$(pgrep -d, -f 'nginx: worker process')" 1 10
findmnt -T /path/to/static
df -h /var/log/nginx

Replace /path/to/static with the real static directory. Static files are often served from page cache, so disk busy alone does not prove they are the bottleneck. Log writes, temporary files and storage latency must be correlated with request latency windows.

Optimization Plan: Change One Config Group at a Time

1. Worker and Event Model

worker_processes auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 16384;
    use epoll;
    multi_accept off;
}

epoll

applies only to Linux; modern Nginx usually selects the optimal event model automatically. worker_rlimit_nofile cannot exceed systemd or kernel limits. multi_accept on may change burst‑accept behavior and should be kept off unless benchmarks prove benefit.

When adjusting systemd limits, create a drop‑in unit rather than editing the distro unit directly:

[Service]
LimitNOFILE=65535

After creating or modifying the drop‑in, run systemctl daemon-reload, then reload/restart Nginx, verifying current fd usage, limits, memory, and baseline metrics before and after.

2. Client and Upstream Keepalive

http {
    keepalive_timeout 65;
    keepalive_requests 1000;
}

upstream app_backend {
    server 10.30.2.21:8080 max_fails=3 fail_timeout=10s;
    server 10.30.2.22:8080 max_fails=3 fail_timeout=10s;
    keepalive 64;
}

server {
    location / {
        proxy_pass http://app_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

The keepalive 64 directive caches idle upstream connections per worker; it does not increase upstream concurrency. Verify upstream supports HTTP/1.1 reuse and monitor upstream_connect_time and errors before enabling.

3. Buffering and Temporary Files

location /api/ {
    proxy_pass http://app_backend;
    proxy_buffering on;
    proxy_buffers 8 16k;
    proxy_buffer_size 16k;
    proxy_busy_buffers_size 32k;
    proxy_max_temp_file_size 1024m;
}

These values are not defaults; buffering increases memory per concurrent request, and large responses may spill to temporary files. Check actual temporary paths, disk space and inode availability:

sudo nginx -T | rg 'proxy_temp_path|client_body_temp_path|fastcgi_temp_path'
df -h /path/to/nginx-temp
df -i /path/to/nginx-temp

For SSE, streaming or partial downloads, do not disable buffering globally; proxy_buffering off; forces slow clients to hold upstream connections. Upload endpoints need separate evaluation of client_max_body_size, client_body_buffer_size and temp directories.

4. Log, Static Files, Compression and Protocol

Access logs can become high‑frequency I/O; they should not be disabled indiscriminately because they may be required for audit and troubleshooting. When health checks do not need logging, use a conditional map:

map $request_uri $loggable {
    default 1;
    =/healthz 0;
}
access_log /var/log/nginx/access.log combined if=$loggable;

Static‑file serving settings (caching metadata and fd):

sendfile on;
 tcp_nopush on;
 open_file_cache max=10000 inactive=30s;
 open_file_cache_valid 60s;
 open_file_cache_min_uses 2;
 open_file_cache_errors on;

Enable gzip cautiously, as it consumes CPU and should not be applied to already compressed formats:

gzip on;
 gzip_min_length 1024;
 gzip_comp_level 4;
 gzip_types text/plain text/css application/json application/javascript application/xml text/xml;
 gzip_vary on;

HTTP/2 enablement varies by Nginx version; older versions require listen ... http2;, newer versions accept a separate http2 on;. Always verify with nginx -t and consult the version‑specific documentation.

Gray‑Scale Deployment, Metric Comparison and Rollback

Modify only one configuration group per iteration. Standard steps:

Divert traffic from a single instance or a small traffic slice.

Save current configuration, version, baseline commands and metrics.

Run sudo nginx -t to validate syntax.

Apply changes with sudo systemctl reload nginx following the organization’s rollout process.

Observe a representative monitoring window; if metrics meet the target, gradually expand the rollout.

In Kubernetes, avoid manual in‑container edits; use images, ConfigMaps, Helm or other declarative methods and verify the correct namespace:

kubectl -n <namespace> rollout status deployment/<deployment-name>
kubectl -n <namespace> get pods -l app=<label-value>

When comparing before and after, keep the request set, duration, load‑test client resources and traffic window identical. Compare success count, status codes, P50/P95/P99 latency, worker CPU/RSS/fd, active connections, upstream response times, soft IRQs, NIC drops, TCP retransmits and disk latency. Use Prometheus metrics as exposed by the actual exporters.

Define rollback conditions in advance (error‑rate rise, P99 exceeding SLO, upstream pool saturation, fd spikes, or security check failures). To roll back, restore the previous configuration, run nginx -t, reload, and verify health checks, core business functionality, error logs and metrics. If systemd or kernel parameters were also changed, revert them separately.

The final deliverable is not a “magic parameter list” but a capacity conclusion that is explainable, gray‑scale‑tested, verifiable and rollback‑ready for the specific workload.