From 5s to 500ms: 12 Nginx Parameters for Performance Tuning

Breaking Down "Slow": Using Timing Fields and Logs to Identify Layers

The article begins by emphasizing that a 5-second to 500ms improvement is a business goal, not guaranteed by simply increasing a few Nginx directives. Total page load time includes DNS, TCP, TLS, browser queue, static resources, Nginx queuing, upstream application, database, and network round-trips. Nginx tuning only eliminates proven web-layer bottlenecks: file descriptor limits, connection model mismatches, inefficient static file delivery, unreasonable compression, or insufficient request buffering. It cannot fix slow SQL, downstream APIs, or cross-region latency.

To diagnose, the author provides a curl command to measure connection, TLS, TTFB, and total times:

curl -sS -o /dev/null -w 'code=%{http_code} connect=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s size=%{size_download}
' <site-url>

A bash script runs 20 serial requests to collect baseline samples, saving output to a timestamped file. The author warns this is not a load test and results must be correlated with access logs, APM, CPU, I/O, and connection counts across low and peak traffic windows.

Additional diagnostic commands:

sudo nginx -V          # version and compiled modules
sudo nginx -T > /tmp/nginx-effective-config.txt  # expanded config with syntax check
ps -C nginx -o pid,ppid,ni,stat,%cpu,%mem,etime,args
ss -s
cat /proc/$(pgrep -o nginx)/limits | rg 'open files'

These reveal master/worker counts, socket summary, and the master's inherited file descriptor limit. If worker CPU is low but upstream_response_time is high, the root cause lies in the application or dependencies, not Nginx connection capacity. Only when error logs show "too many open files" should nofile and connection limits be examined.

To separate Nginx queuing from upstream processing, a custom log format with timing fields is introduced:

log_format timing '$remote_addr $host "$request" $status $body_bytes_sent '
                'request_time=$request_time '
                'upstream_connect_time=$upstream_connect_time '
                'upstream_header_time=$upstream_header_time '
                'upstream_response_time=$upstream_response_time';
access_log /var/log/nginx/access-timing.log timing;

request_time

covers Nginx from first request byte to last response byte; upstream fields exist only when proxying. Static files naturally lack upstream times. Slow requests must be segmented by URI, status code, response size, cache hit, and upstream group — averages cannot replace percentiles.

An awk one-liner provides directional statistics (count, average, requests over 1s) but does not compute p95/p99 or prove root cause. Cross-referencing raw logs, system monitoring, and upstream monitoring is essential; if upstream_response_time degrades, optimizing gzip or keepalive merely masks the problem.

Preparing Backup, Canary, and Rollback Before Configuration Changes

Performance changes alter site-wide connection behavior. The author stresses verifying configuration management (Ansible, GitOps, Helm), ensuring at least one canary instance, functional load balancer draining, health checks, and clear error logs. Without a canary channel, reproduce in staging; never reload all nodes simultaneously.

A backup script preserves the entire /etc/nginx directory, effective configuration ( nginx -T), service status, and SHA256 checksums to a controlled directory (e.g., /var/backups/nginx-tune-20260730). cp -a retains permissions, timestamps, and symlinks for emergency rollback. If configuration originates from version control or automation, the true rollback point is the corresponding commit or release; directory copies must not bypass managed workflows. Backups must not be placed in web roots or shared with unprivileged users.

Parameter 1: worker_processes

Determines worker count. On Linux, auto lets Nginx choose based on visible CPUs. In containers, CPU quotas, cpuset, and NUMA environments, the actual result must be observed via process count and cgroup — do not blindly set a large number matching physical cores.

# nginx.conf main context
worker_processes auto;

Adding workers does not speed up slow upstreams; it increases connections, buffers, and context switches. Start with auto; only investigate worker count and affinity when CPU is saturated, run queue grows, and upstream times are normal. After changes, observe request latency, CPU steal, context switches, and per-worker balance.

nproc
ps -C nginx -o pid,ppid,psr,stat,%cpu,cmd
grep -E 'Cpus_allowed_list|Mems_allowed_list' /proc/$(pgrep -o nginx)/status

psr

shows recent CPU, not permanent binding. Container nproc often differs from host cores; plan by actual visible CPUs. Without evidence, fixing worker_processes risks issues during scaling or node migration.

Parameter 2: worker_cpu_affinity

Binds workers to CPUs via bitmask. Not a default requirement: modern Linux schedulers handle typical web loads; incorrect masks or container CPU set changes can cause contention. Use only on fixed cores, with visible scheduling migration, and reproducible benchmarks.

# Example for 4 visible CPUs; mask order must match actual CPU set.
worker_processes 4;
worker_cpu_affinity 0001 0010 0100 1000;

This snippet cannot be directly applied to hyper-threading, NUMA, Kubernetes CPU Manager, or dynamic cpuset. Before applying, verify systemd CPUAffinity, container runtime, and cgroup limits. After canary, check per-worker CPU balance and tail latency improvement. Without measurable benefit, remove and revert to auto.

Use taskset -pc to read a worker's affinity mask (does not change binding). If results mismatch expectations, check systemd unit, container CPU set, and boot parameters — do not substitute a temporary taskset for auditable long-term config.

Parameter 3: worker_rlimit_nofile

Each client connection, upstream connection, log file, and cache file consumes a file descriptor. worker_rlimit_nofile raises the worker's attempted limit but cannot exceed systemd or shell hard limits. If error logs show "too many open files", check all three limit layers and actual FD usage.

# nginx.conf main context; value must not exceed system hard limit.
worker_rlimit_nofile 65535;

65535 is an example, not a capacity planning conclusion. If projected concurrency, upstream sockets, logs, and cache files are well below current limits, raising the value will not reduce latency. If increased, systemd LimitNOFILE must match, otherwise new workers inherit the lower limit.

# /etc/systemd/system/nginx.service.d/limits.conf
[Service]
LimitNOFILE=65535

After modifying the drop-in, run daemon-reload, then nginx -t and canary reload. daemon-reload does not restart the service, but reload creates new workers; long-lived connections, SSE, WebSocket, and downloads must be observed during the window. If Nginx is at peak, validate on a low-risk instance first.

sudo systemctl daemon-reload
sudo nginx -t
sudo systemctl reload nginx
cat /proc/$(pgrep -o nginx)/limits | rg 'open files'

If nginx -t fails, do not reload; fix the file or restore backup and re-check. If limits remain unchanged after reload, check the override chain with systemctl show nginx -p LimitNOFILE instead of repeatedly raising worker_rlimit_nofile.

Parameter 4: worker_connections

Maximum simultaneous connections per worker. In reverse proxy, one request typically consumes at least two connections (client + upstream), so worker_processes × worker_connections is not the servable request count. It is also constrained by FD, memory, TLS, keepalive ratio, and upstream connection limits.

events {
    worker_connections 4096;
}

4096 is an example; calculate based on peak connections, growth headroom, and available file descriptors. Raising it may let Nginx accept more slow clients; if upstream, memory, and bandwidth lack headroom, it only spreads queuing and failure. Evidence must come from error logs, listen queues, and connection states.

ss -Htan state established | wc -l
ss -Htan state time-wait | wc -l
sudo rg -n 'worker_connections are not enough|too many open files' /var/log/nginx/error.log

Connection counts must be collected continuously at peak; instantaneous numbers do not represent capacity. High TIME_WAIT often relates to short connections — prioritize client reuse, proxy strategy, and upstream keepalive; do not tweak dangerous kernel parameters just to clear TIME_WAIT.

Parameter 5: multi_accept

multi_accept on

lets a worker accept as many new connections as possible per event loop iteration. It may help short-connection bursts and listen queue backlogs, but can also cause a single worker to grab too many connections, creating imbalance. Steady traffic, HTTP/2, or many long connections may not benefit.

events {
    multi_accept off;
}

Keep explicitly off; only consider on after confirming new-connection bursts, listen queue buildup, and canary improvement. Load balancer health check flapping, client retry storms, and upstream timeouts are often the real burst causes — do not adjust accept strategy alone.

Use ss -ltn sport = :443 and ss -s to read listen socket queue and summary state. Recv-Q and Send-Q provide queuing clues, but meanings vary by kernel version and port reuse model; compare across similar traffic windows, not a single screenshot.

Parameter 6: use epoll

On Linux, Nginx usually auto-selects epoll. Explicit use epoll mainly aids config clarity or interference exclusion, not a universal speed switch. Forcing it on non-Linux, unsupported builds, or conflicting configs can cause startup failure.

events {
    use epoll;
}

Prerequisite: Linux and current Nginx build support the event model. On startup or reload anomalies, remove the directive to restore auto-selection. Event model and upstream slowness are separate issues; do not attribute backend blocking to epoll.

Confirm via nginx -T and nginx.service startup logs whether the directive loads and produces errors. Without log and metric evidence, keeping default auto-selection is safer.

Parameter 7: keepalive_timeout

HTTP keep-alive reduces TCP/TLS rebuilds, but overly long idle timeouts let slow clients occupy workers and FDs; too short increases handshakes and TIME_WAIT. This value must coordinate with client types, upstream load balancer idle timeout, CDN, and HTTP/2 connection model.

http {
    keepalive_timeout 65s;
}

65 seconds is a common default example; do not infinitely increase because "long connections are good." If the front LB has a shorter idle timeout, a longer Nginx setting yields no benefit. After changes, monitor established, TIME_WAIT, client 499/502/504, TLS handshake rate, and connection lifetime — especially protect long-polling and SSE.

sudo nginx -T 2>&1 | rg -n 'keepalive_timeout|keepalive_requests'
ss -Htan state established '( sport = :80 or sport = :443 )' | wc -l

The second command is a snapshot; not all connections are idle. If connections are high but throughput low and FD near limit, shortening timeout may be more effective than raising worker_connections; still canary to avoid disrupting mobile and proxied slow requests.

Parameter 8: keepalive_requests

Limits maximum requests per client connection. Moderately increasing reduces frequent reconnects; too low increases handshakes and TIME_WAIT; too high lets long-lived connections hog resources and cause client load imbalance. HTTP/2 multiplexing behaves differently from HTTP/1.1 — do not copy values blindly.

http {
    keepalive_requests 1000;
}

1000 is an example; validate against request size, client type, and connection lifetime. If TLS handshakes, client reconnects, or abnormal status codes rise after change, revert. Any request-level metrics must be isolated from releases and traffic structure changes to avoid attributing natural fluctuations to this parameter.

Track 5xx trends in log platform using actual access log format. Default combined log's ninth field is status code; custom formats need corresponding field adjustment. Status code trends alone cannot prove keepalive_requests is root cause — simultaneously check request volume, client errors, and connection metrics.

Parameter 9: sendfile with tcp_nopush

sendfile on

lets Nginx send local static files via kernel path, suitable for disk-based static assets. tcp_nopush works with sendfile to coalesce response headers and large file chunks. They are not equivalent to dynamic API acceleration; for NFS, FUSE, frequent atomic replacement, or consistency-sensitive paths, pre-production validation is mandatory.

http {
    sendfile on;
    tcp_nopush on;
}

Before enabling, confirm resources come from a reliable local filesystem. After change, verify Content-Length, Range, download integrity, cache hits, and large file responses; if stale reads, range request anomalies, or download failures appear, revert first, then check storage mounts and deployment methods.

curl -sS -D /tmp/headers.txt -o /dev/null -r 0-1023 <static-file-url>
rg -n 'HTTP/|Content-Length|Content-Range|Accept-Ranges' /tmp/headers.txt

Replace <static-file-url> with a real, non-sensitive large static resource. Range only checks protocol response, not full file content; artifact integrity still relies on the release system's checksums.

Parameter 10: tcp_nodelay

tcp_nodelay on

disables Nagle's algorithm, reducing latency for small responses waiting for coalescing. Typically suits APIs, small responses, and interactive requests; impact on large file throughput is less pronounced than sendfile path. It does not fix packet loss, congestion, or upstream blocking.

http {
    tcp_nodelay on;
}

Typical web services can start with on, but still measure TTFB, packet count, and CPU on canary instances. Coexistence with tcp_nopush is not contradictory: the former prioritizes immediate send of small responses, the latter serves large file blocks with sendfile. If dynamic API upstream time dominates, transport parameters should not preempt application optimization priority.

Validation should use the initial curl timing command on the same URL, similar network location, and comparable load. Single fluctuations do not prove latency improvement; for dynamic APIs, also cross-check access log upstream_response_time.

Parameter 11: client_header_buffer_size and large_client_header_buffers

Request header buffers determine Nginx's capacity to read URI, Cookie, Authorization, etc. Too small yields 400 or 414; too large raises memory under high concurrency and amplifies resource consumption of oversized header requests. It is not a license for unlimited Cookie growth.

http {
    client_header_buffer_size 1k;
    large_client_header_buffers 4 8k;
}

Example suits 64-bit Linux but must be validated against actual Cookie, JWT, SSO header lengths and Nginx build. On 400/414, first confirm client sent too long header line in error log, then identify whether business Cookie bloat, proxy-added headers, or anomalous requests are the cause — do not simply enlarge buffers.

Filter Nginx error log for too long header, client sent too large request, 414, and 400. Log matching is part of root cause evidence; if cause is oversized JWT or Cookie, long-term fix is reducing session payload, moving to server-side sessions, or fixing proxy header stacking — not permanently enlarging all workers' header buffers. After increasing, monitor worker RSS and anomalous request ratio.

Parameter 12: gzip

Gzip compresses text responses in Nginx, reducing public network bytes but consuming CPU. Already-compressed types (JPEG, PNG, MP4, ZIP) gain almost nothing. Before enabling, confirm CDN, upstream, and reverse proxies are not already compressing, to avoid double compression, cache variant confusion, or CPU starvation.

http {
    gzip on;
    gzip_comp_level 4;
    gzip_min_length 1024;
    gzip_vary on;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/javascript application/xml image/svg+xml;
}

gzip_types

omits text/html because Nginx compresses HTML by default. Level 4 is a CPU/compression trade-off example, not a fixed optimum. gzip_proxied any requires review with caching and response header strategy. Do not add application/zip, image/jpeg, video/mp4 to the list.

curl -sS -H 'Accept-Encoding: gzip' -D /tmp/gzip-headers.txt -o /dev/null <site-url>
rg -n -i 'content-encoding|vary|content-length|cache-control' /tmp/gzip-headers.txt

Verify content correctness, Content-Encoding appears only after negotiation, Vary includes Accept-Encoding, CDN cache hit changes. If CPU is already high or upstream already compresses, disabling Nginx gzip may be better; judge by end-to-end TTFB, download time, and CPU, not just byte count.

One Auditable Canary Change

The article provides an example include file ( /etc/nginx/conf.d/20-performance.conf) containing only baseline-supported directives — not a "twelve parameters all on" template. If original config defines same directives elsewhere, modify original location or remove duplicates to avoid nginx -t failure.

# /etc/nginx/conf.d/20-performance.conf
# Prerequisite: this file is indeed included by nginx.conf http context.
keepalive_timeout 65s;
keepalive_requests 1000;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
gzip on;
gzip_comp_level 4;
gzip_min_length 1024;
gzip_vary on;
gzip_proxied any;
gzip_types text/plain text/css application/json application/javascript application/xml image/svg+xml;

Configuration changes must follow: backup → modify → syntax check → canary apply → validate → rollback. Reload is gentler than restart but still replaces workers; long connections, WebSocket, SSE, and large downloads need drain observation. Without idle canary instances, test in staging or add a new node.

Apply script:

#!/usr/bin/env bash
set -euo pipefail
config_file="/etc/nginx/conf.d/20-performance.conf"
backup_dir="<backup-dir>"
sudo install -d -m 0700 "$backup_dir"
sudo cp -a "$config_file" "$backup_dir/20-performance.conf.before"
sudo nginx -t
sudo systemctl reload nginx
sleep 2
sudo systemctl is-active --quiet nginx
curl -fsS -o /dev/null <health-check-url>

curl

success only proves checkpoint reachability; must also verify key pages, APIs, static files, TLS, error rates, and access/error logs. If nginx -t fails, stop, restore file, re-check. If health check fails after reload, rollback instead of stacking more parameters.

Rollback script:

#!/usr/bin/env bash
set -euo pipefail
config_file="/etc/nginx/conf.d/20-performance.conf"
backup_file="<backup-dir>/20-performance.conf.before"
sudo cp -a "$backup_file" "$config_file"
sudo nginx -t
sudo systemctl reload nginx
curl -fsS -o /dev/null <health-check-url>

Rollback assumes backup exists and belongs to current instance. Also reloads; execute within change window and on canary node. If Nginx fails to start, preserve error logs first, then follow service recovery process. Do not trial-by-deleting config files because include chain may still reference them.

Proving Tuning Effectiveness

An effective tuning compares at least two metric categories: (1) same URL set TTFB, total, status codes, response size; (2) Nginx 4xx/5xx, request_time, upstream_response_time; (3) worker CPU/RSS, FD, established/TIME_WAIT, network retransmits, disk I/O. Prometheus metric names and labels depend on actual exporter; do not fabricate metrics in dashboards if exporter absent.

# Based on actual exporter: Nginx process open FD usage ratio.
process_open_fds{job="nginx"}
/
process_max_fds{job="nginx"}

# Based on actual exporter: node established TCP connection trend.
node_netstat_Tcp_CurrEstab

The first relies on process exporter or equivalent; the second often from node_exporter's netstat collector — actual deployment may lack or rename them; verify in Targets and /metrics before use. Final conclusion must be verifiable fact, e.g., "canary window saw no FD exhaustion errors, connection usage below planned ceiling, and tail latency of same request set did not regress"; not "changed worker_connections so site is definitely faster." If upstream remains slow, pivot to application thread pools, database, cache, and dependency calls — not stacking more Nginx parameters.

Parameters interact; review must prevent "locally correct, globally degraded." Example: raising worker_connections without raising FD limit merely hits process limit sooner; extending keepalive_timeout reduces handshakes but may let slow clients hold FD and memory longer; enabling gzip shrinks text responses but may raise TTFB when CPU is tight; sendfile suits local static files, not proxy API optimization. Change only one hypothesis-related parameter group at a time to establish causality from before/after metrics.

Distinguish Nginx request time from user-perceived load time. First paint depends on HTML, critical CSS, JS, images, fonts, and browser cache. If HTML TTFB is stable but page remains slow, inspect resource waterfall, cache headers, HTTP/2 multiplexing, CDN hits, frontend bundle size, third-party scripts, and network round-trips. Further squeezing worker parameters yields diminishing returns and makes config unmaintainable. Record Nginx changes, app releases, cache invalidations, and traffic shifts on a shared timeline to avoid misattributing natural fluctuations to a config change.