Cut Page Load from 5 s to 500 ms: 12 Essential Nginx Performance Tweaks

Reducing page load time from 5 seconds to 500 ms requires analyzing the full request path—DNS, TCP/TLS handshakes, browser queuing, static‑file delivery, Nginx queuing, upstream processing, database calls, and network round‑trips. Nginx tuning is useful only when evidence shows bottlenecks in the web layer such as file‑descriptor exhaustion, mismatched connection models, inefficient static‑file paths, unreasonable compression, or insufficient request buffering.

Measurement Foundations

Use curl to capture timing fields for the target HTTPS URL (e.g., https://www.example.com/health):

curl -sS -o /dev/null -w 'code=%{http_code} connect=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s size=%{size_download}
' https://www.example.com/health

Repeat the measurement in low‑peak and high‑peak windows and correlate the results with access logs, APM data, CPU/I/O stats, and connection counts.

Baseline Sampling Script

#!/usr/bin/env bash
set -euo pipefail
url="https://www.example.com/health"
count=20
for i in $(seq 1 "$count"); do
  curl -sS -o /dev/null -w '%{time_connect} %{time_starttransfer} %{time_total} %{http_code}
' "$url"
done | tee "/tmp/nginx-curl-sample-$(date +%Y%m%d%H%M%S).txt"

This low‑frequency script provides a baseline; it is not a load‑test tool. Preserve the raw output for later analysis.

Inspecting the Effective Configuration

sudo nginx -V
sudo nginx -T > /tmp/nginx-effective-config.txt

-V

shows compiled modules (e.g., http_v2, stub_status, thread_pool). -T dumps the full configuration after processing all include directives, allowing verification that the intended file is loaded.

Key Metrics to Observe

Master/worker process count and CPU affinity: ps -C nginx Open‑file limits: cat /proc/$(pgrep -o nginx)/limits | grep 'open files' Connection states: ss -s and per‑port counts

If the error log contains too many open files, investigate both system‑wide and Nginx‑specific limits before increasing worker_connections.

Precise Timing Log Format

log_format timing '$remote_addr $host "$request" $status $body_bytes_sent '
'request_time=$request_time '
'upstream_connect_time=$upstream_connect_time '
'upstream_header_time=$upstream_header_time '
'upstream_response_time=$upstream_response_time';
access_log /var/log/nginx/access-timing.log timing;

This format records request latency, upstream connection time, header time, and response time, enabling per‑URI, status‑code, and size breakdowns.

Parameter 1 – worker_processes

Set worker_processes auto; so Nginx matches the visible CPU count. In containers or NUMA environments verify the actual CPU set with nproc and ps -C nginx -o psr. Increase only when CPU is saturated and upstream latency is normal.

# nginx.conf main context
worker_processes auto;

Parameter 2 – worker_cpu_affinity

Bind workers to specific CPUs with a bitmask (e.g., 0001 0010 0100 1000 for four CPUs). Use only when the scheduler shows clear contention; otherwise the default scheduler is sufficient.

worker_processes 4;
worker_cpu_affinity 0001 0010 0100 1000;

Parameter 3 – worker_rlimit_nofile

Raise the hard limit for file descriptors that workers can set, matching the systemd LimitNOFILE value. Example:

worker_rlimit_nofile 65535;

If the error log reports “too many open files”, also adjust the systemd drop‑in:

# /etc/systemd/system/nginx.service.d/limits.conf
[Service]
LimitNOFILE=65535

Parameter 4 – worker_connections

Defines the maximum simultaneous connections per worker. A typical example is worker_connections 4096;, but the real limit must consider the total file‑descriptor ceiling and expected peak connections.

events {
    worker_connections 4096;
}

Parameter 5 – multi_accept

When multi_accept off, a worker accepts one connection per event loop, which avoids imbalance under steady load. Enable on only after observing queue buildup and confirming improvement.

events {
    multi_accept off;
}

Parameter 6 – use epoll

Linux automatically selects epoll. Explicitly setting use epoll; is mainly for clarity; remove it if the build does not support the event model.

events {
    use epoll;
}

Parameter 7 – keepalive_timeout

Controls how long idle keep‑alive connections stay open. The common default is 65s. Align this value with upstream idle time; a longer timeout reduces handshakes but can tie up workers and file descriptors.

http {
    keepalive_timeout 65s;
}

Parameter 8 – keepalive_requests

Limits the number of requests per keep‑alive connection. Example: keepalive_requests 1000;. Adjust based on request size and client behavior; monitor for increased 4xx/5xx if connections linger too long.

http {
    keepalive_requests 1000;
}

Parameter 9 – sendfile & tcp_nopush

Enable kernel‑direct static file delivery with sendfile on; and improve packet framing with tcp_nopush on;. Verify on local files; test with range requests to ensure correct Content‑Length and Content‑Range headers.

http {
    sendfile on;
    tcp_nopush on;
}

Parameter 10 – tcp_nodelay

Disable Nagle’s algorithm for small responses: tcp_nodelay on;. Useful for APIs and interactive traffic; confirm improvement with TTFB measurements.

http {
    tcp_nodelay on;
}

Parameter 11 – client_header_buffer_size & large_client_header_buffers

Set reasonable header buffers, e.g., client_header_buffer_size 1k; and large_client_header_buffers 4 8k;. Increase only after observing 400/414 errors caused by oversized cookies or JWTs.

http {
    client_header_buffer_size 1k;
    large_client_header_buffers 4 8k;
}

Parameter 12 – gzip

Compress textual responses with gzip on; and moderate settings ( gzip_comp_level 4;, gzip_min_length 1024;, gzip_vary on;, appropriate gzip_types). Ensure upstream services or CDNs are not already compressing the same content.

http {
    gzip on;
    gzip_comp_level 4;
    gzip_min_length 1024;
    gzip_vary on;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/javascript application/xml image/svg+xml;
}

Gray‑Scale Deployment Workflow

Before applying any change, back up the current configuration, run nginx -t for syntax checking, reload the service on a single gray instance, and verify health checks, error logs, and key metrics (TTFB, request_time, CPU, FD usage). If the reload fails, restore the backup and retry.

#!/usr/bin/env bash
set -euo pipefail
backup_dir="/var/backups/nginx-tune-20260730"
install -d -m 0700 "$backup_dir"
cp -a /etc/nginx "$backup_dir/etc-nginx-before"
nginx -T > "$backup_dir/nginx-T-before.txt"
systemctl status nginx --no-pager > "$backup_dir/nginx-status-before.txt"
sha256sum "$backup_dir/nginx-T-before.txt" > "$backup_dir/nginx-T-before.sha256"

Verification Metrics

Compare two sets of metrics for the same URL collection:

TTFB, total time, status codes, response size

Nginx 4xx/5xx, request_time, upstream_response_time Worker CPU/RSS, open‑file usage, established/TIME_WAIT connections, network retransmits, disk I/O

Example Prometheus queries (adjust names/labels to match your exporters):

# Nginx process file‑descriptor usage ratio
process_open_fds{job="nginx"} / process_max_fds{job="nginx"}
# Current established TCP connections
node_netstat_Tcp_CurrEstab

Common Pitfalls

Increasing worker_connections without raising worker_rlimit_nofile can cause earlier FD exhaustion.

Extending keepalive_timeout may reduce handshakes but can let slow clients hold workers and file descriptors.

Enabling gzip on a CPU‑bound server may increase TTFB; disable if upstream already compresses.

Using sendfile for proxied API traffic provides no benefit and can hide upstream latency.

Apply one related group of parameters at a time, observe before/after metrics, and only then proceed to the next hypothesis.