Mastering Nginx Under High Load: Practical Tuning & Troubleshooting Guide

Nginx is renowned for handling massive concurrency with low memory usage, so performance problems usually stem from the surrounding system architecture, OS configuration, or backend services rather than Nginx itself.

1️⃣ Operating‑System Layer – Most Common Bottlenecks

Default OS limits are insufficient for extreme traffic. Increase file descriptor limits and adjust kernel parameters.

# /etc/sysctl.conf
fs.file-max = 1000000

# /etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535

Also raise the per‑process limit inside nginx.conf:

worker_rlimit_nofile 65535;

2️⃣ Network‑Stack Optimisation

Prevent TCP port exhaustion and reduce TIME_WAIT sockets.

# Expand local port range
net.ipv4.ip_local_port_range = 1024 65535

# Reuse TIME_WAIT sockets and shorten timeout
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30

# Increase socket buffers and backlog
net.core.somaxconn = 65535
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

3️⃣ Nginx Configuration Layer

Set appropriate worker processes and connections, and optimise logging.

# nginx.conf
worker_processes auto;

events {
    worker_connections 65535;
    use epoll;  # Linux
}

# Reduce log I/O overhead
access_log /var/log/nginx/access.log main buffer=64k flush=1m;

During peak periods you can temporarily disable access_log to avoid I/O bottlenecks.

4️⃣ Reverse‑Proxy Optimisation

Enable keep‑alive and connection reuse for upstream servers.

upstream backend_servers {
    server 10.1.2.3:8080;
    server 10.1.2.4:8080;
    keepalive 64;
}

server {
    location / {
        proxy_pass http://backend_servers;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

5️⃣ Backend Service Layer – The Real Bottleneck

If backend processing is slow, Nginx connections remain occupied, causing worker_connections to fill up. Mitigate by:

Optimising SQL queries, adding caches (Redis/Memcached), and refactoring business logic.

Scaling out backend servers behind a load balancer.

Adjusting proxy timeouts such as proxy_read_timeout and proxy_connect_timeout.

Horizontal or vertical scaling of backend resources.

6️⃣ Hardware & Infrastructure Layer

Insufficient CPU cores, memory, bandwidth, or reliance on mechanical disks can limit throughput. Use monitoring tools like htop, iftop, and iotop to identify resource saturation and plan upgrades.

7️⃣ Systematic Troubleshooting Workflow

Monitor: nginx-status, netstat, ss, vmstat, dstat.

Inspect logs: check error.log for clues.

Top‑down investigation:

OS: ulimit -n, ss -s, netstat -s | grep timestamp Nginx config: verify worker_processes, worker_connections, logging, keep‑alive settings.

Backend: use load‑testing tools like wrk, ab, jmeter to measure real throughput.

⚡ Summary

Nginx itself is a robust front‑door; most performance issues lie in the operating system or backend applications.

By correctly tuning the OS, Nginx parameters, backend services, and hardware resources, you can fully unleash Nginx’s high‑concurrency capabilities.