Tuning Nginx Worker Processes and Connections for High‑Concurrency Scenarios

Problem Background

When Nginx serves as the entry gateway under heavy traffic, mis‑configured parameters can cause connection refusals, slow responses, uneven CPU usage, or memory exhaustion, potentially leading to a service avalanche.

Core Architecture

Nginx uses a Master‑Worker multi‑process model. The master process reads the configuration, spawns workers, and handles signals, while each worker processes client requests, performs reverse proxy, serves static files, and handles FastCGI/uWSGI.

Workers operate on an event‑driven, asynchronous non‑blocking I/O model (epoll on Linux, kqueue on BSD), allowing a single worker to manage tens of thousands of concurrent connections.

Key Parameters and Their Meaning

worker_processes : Number of worker processes, usually set to the number of CPU cores (or auto).

worker_cpu_affinity : Binds each worker to a specific CPU core to avoid context‑switch overhead.

worker_connections : Maximum simultaneous connections a worker can handle. The total possible connections are worker_processes * worker_connections. For reverse‑proxy mode, the effective maximum is half of that because each client‑backend pair uses two file descriptors.

worker_rlimit_nofile : Upper limit of file descriptors a worker may open; must be ≥ worker_connections.

multi_accept (on/off): Accept multiple connections per event to reduce epoll calls.

accept_mutex (on/off): Mutex for connection acceptance; modern kernels often disable it ( off) because the “thundering herd” problem is mitigated.

keepalive_timeout and keepalive_requests : Control HTTP keep‑alive behavior.

sendfile , tcp_nopush , tcp_nodelay : Zero‑copy file transfer and TCP optimizations.

open_file_cache : Caches static‑file metadata to reduce open() system calls.

upstream keepalive : Number of persistent connections from Nginx to backend servers.

System‑Level Tuning

Increase the OS file‑descriptor limit and adjust TCP kernel parameters:

# /etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535

# /etc/sysctl.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.ip_local_port_range = 10000 65535

Apply the changes with sysctl -p and verify using ulimit -n and sysctl -a | grep somaxconn.

Practical Configuration Example

Below is a minimal nginx.conf that follows the recommended defaults for a high‑concurrency deployment:

worker_processes auto;
worker_cpu_affinity auto;
worker_rlimit_nofile 65535;

events {
    use epoll;
    worker_connections 10240;
    multi_accept on;
    accept_mutex off;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    keepalive_requests 100;
    client_body_buffer_size 128k;
    client_max_body_size 10m;
    open_file_cache max=10000 inactive=60s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;
    server_tokens off;
    include /etc/nginx/conf.d/*.conf;
}

For a reverse‑proxy scenario, define an upstream block with keepalive and appropriate health‑check parameters:

upstream backend {
    least_conn;
    server 192.168.1.10:8080 weight=5 max_fails=3 fail_timeout=30s;
    server 192.168.1.11:8080 weight=3 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

server {
    listen 80;
    server_name api.example.com;
    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 10s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
        proxy_buffer_size 4k;
        proxy_buffers 8 4k;
        proxy_busy_buffers_size 8k;
        proxy_next_upstream error timeout http_502 http_503 http_504;
    }
}

Step‑by‑Step Optimization Workflow

Identify business metrics: peak concurrent connections, expected QPS, request mix (static vs dynamic).

Assess hardware: CPU cores ( nproc), memory, network bandwidth.

Set worker_processes to the core count.

Calculate worker_connections based on the desired total connections (e.g., total = 200,000 → worker_connections = total / worker_processes).

Adjust OS limits ( ulimit, limits.conf) and TCP parameters ( sysctl.conf).

Enable performance‑enhancing options ( sendfile, tcp_nopush, tcp_nodelay, open_file_cache).

Reload Nginx ( nginx -t && nginx -s reload) and verify with curl http://127.0.0.1/nginx_status.

Run load tests (e.g., ab -n 10000 -c 1000 http://example.com/ or wrk -t4 -c1000 -d30s http://example.com/) and record QPS, latency, error rate.

Monitor system resources ( top, htop, vmstat 1) and Nginx metrics (stub_status, Prometheus exporter).

Iteratively tune parameters based on observed bottlenecks (increase worker_connections, enable worker_cpu_affinity, adjust keepalive_timeout, etc.).

Monitoring and Alerting

Enable stub_status for quick health checks and use nginx‑prometheus‑exporter for detailed metrics. Example Prometheus alerts:

groups:
- name: nginx
  rules:
  - alert: NginxHighConnections
    expr: nginx_connections_active / (worker_connections * worker_processes) > 0.8
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Nginx active connections high"
      description: "Active connections are {{ $value }}% of max"
  - alert: NginxHighRequestRate
    expr: rate(nginx_http_requests_total[5m]) > 10000
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Nginx request rate high"
      description: "Request rate is {{ $value }} req/s"

Risk Mitigation and Rollback

Before any change, back up nginx.conf and sysctl.conf. Test the new configuration with nginx -t. If problems arise, restore the backup and reload:

cp /etc/nginx/nginx.conf.bak /etc/nginx/nginx.conf
nginx -t && nginx -s reload

For system‑level changes, replace sysctl.conf with the backup and re‑apply:

cp /etc/sysctl.conf.bak /etc/sysctl.conf
sysctl -p

Production Checklist

Backup configs and document the change plan.

Perform changes during low‑traffic windows.

Notify stakeholders and have a clear rollback command ready.

Validate with nginx -t, curl health check, and a quick load test.

Continuously monitor CPU, memory, file‑descriptor usage, and Nginx metrics for at least 30 minutes after the change.

Update operational runbooks with new parameter values and observed performance gains.

Summary

Effective high‑concurrency tuning of Nginx consists of aligning worker_processes with CPU cores, sizing worker_connections to meet expected concurrent connections, raising OS file‑descriptor limits, enabling zero‑copy and TCP optimizations, configuring appropriate keep‑alive and caching settings, and validating the result with systematic load testing and monitoring. By following the step‑by‑step workflow, operators can prevent service degradation, achieve higher QPS, and maintain stability under traffic spikes.