Tune Nginx for Million‑PPS: Kernel & Config Optimizations

Tune Nginx for Million‑PPS: Kernel & Config Optimizations

Applicable Scenarios and Prerequisites

Application scenarios: high‑concurrency web services, CDN edge nodes, API gateways, reverse proxies.

Prerequisites: Nginx 1.15+, Linux kernel 4.18+, root access, pressure‑test tools (wrk/ab/vegeta).

Performance goals: >100 k concurrent connections, >1 M PPS, P99 latency <50 ms.

Environment and Version Matrix

Component : Nginx – Recommended version 1.24+, Minimum 1.15+ (LTS is more stable).

Kernel : 5.10+ recommended, minimum 4.18+ (new kernels give 10‑20% performance boost).

CPU : 10 GbE NIC recommended; 1 GbE is the minimum (network card can be a bottleneck).

Memory : ≥4 GB recommended, ≥2 GB minimum (buffer/connection pools).

File descriptors : set ulimit -n 100000+, minimum 65535 (limits concurrent connections).

Network stack : tune net.ipv4.* parameters (significant packet loss/latency improvement).

Quick Checklist

Step 1 : Optimize Linux kernel network parameters (sysctl).

Step 2 : Adjust system resource limits (ulimit, file descriptors).

Step 3 : Optimize Nginx worker process model.

Step 4 : Configure efficient I/O multiplexing.

Step 5 : Optimize backend connection reuse and timeout policies.

Step 6 : Enable HTTP/2 and dynamic gzip compression.

Step 7 : Conduct benchmark testing and performance verification.

Step 8 : Build monitoring and alerting system.

Implementation Steps

Step 1: Optimize Linux Kernel Network Parameters

Edit sysctl configuration: sudo vi /etc/sysctl.conf Production‑grade parameters:

# ============ TCP connection parameters ============
net.core.somaxconn = 65535            # increase listen queue
net.ipv4.tcp_max_syn_backlog = 65535   # SYN backlog for SYN‑Flood protection
net.ipv4.tcp_max_tw_buckets = 1000000 # high‑capacity TIME_WAIT
# ============ TIME_WAIT management ============
net.ipv4.tcp_tw_reuse = 1            # allow TIME_WAIT reuse
net.ipv4.tcp_fin_timeout = 30        # FIN timeout seconds
net.ipv4.tcp_tw_recycle = 0          # keep safety checks disabled on trusted LAN
# ============ Connection tracking ============
net.netfilter.nf_conntrack_max = 2000000
net.netfilter.nf_conntrack_tcp_timeout_established = 600
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 60
# ============ Port range ============
net.ipv4.ip_local_port_range = 1024 65535
# ============ TCP keepalive ============
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 30
# ============ TCP window ============
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
# ============ Buffer sizes ============
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_rmem = 4096 65536 16777216
net.ipv4.udp_mem = 94500000 1300000000 2000000000
# ============ Network driver ============
net.core.netdev_max_backlog = 65535
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default = 8388608
net.core.wmem_default = 8388608
# ============ IP forwarding & ICMP ============
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.fib_multipath_hash_policy = 1
# ============ TCP optimizations ============
net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_mtu_probing = 1

Apply the changes:

sudo sysctl -p

Step 2: Adjust System Resource Limits

Check current limits: ulimit -a Temporary (session‑only) increase:

ulimit -n 100000
ulimit -s unlimited

Permanent change for all users (edit /etc/security/limits.conf):

* soft nofile 100000
* hard nofile 100000
* soft nproc 100000
* hard nproc 100000
* soft memlock unlimited
* hard memlock unlimited

Specific Nginx user limits:

nginx soft nofile 100000
nginx hard nofile 100000
nginx soft nproc 65535
nginx hard nproc 65535

Verify after re‑login:

ulimit -n

Step 3: Optimize Nginx Worker Process Model

Edit /etc/nginx/nginx.conf and apply the core settings:

user nginx nginx;
worker_processes auto;
worker_cpu_affinity auto;
worker_priority -10;
worker_rlimit_nofile 100000;

events {
    worker_connections 65535;
    use epoll;
    multi_accept on;
    accept_mutex off;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" rt=$request_time uct=$upstream_connect_time uht=$upstream_header_time urt=$upstream_response_time';
    access_log /var/log/nginx/access.log main buffer=32k flush=5s;
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    client_max_body_size 100m;
    gzip on;
    gzip_min_length 1k;
    gzip_comp_level 6;
    gzip_types text/plain text/css text/xml text/javascript application/json;
    gzip_vary on;
    upstream backend {
        least_conn;
        keepalive 256;
        server 192.168.1.10:8080 max_fails=3 fail_timeout=30s weight=1;
        server 192.168.1.11:8080 max_fails=3 fail_timeout=30s weight=1;
        server 192.168.1.12:8080 max_fails=3 fail_timeout=30s weight=1;
    }
    server {
        listen 80 default_server reuseport backlog=65535;
        listen [::]:80 default_server reuseport;
        server_name _;
        location / {
            proxy_pass http://backend;
            proxy_http_version 1.1;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Connection "";   # clear to enable reuse
            proxy_connect_timeout 10s;
            proxy_send_timeout 30s;
            proxy_read_timeout 30s;
            proxy_buffering on;
            proxy_buffer_size 8k;
            proxy_buffers 8 8k;
            proxy_socket_keepalive on;
        }
        location /health {
            access_log off;
            return 200 "ok
";
            add_header Content-Type text/plain;
        }
        location /nginx_status {
            stub_status on;
            access_log off;
            allow 127.0.0.1;
            deny all;
        }
    }
}

Validate syntax and reload without downtime:

sudo nginx -t
sudo nginx -s reload

Step 4: Verify I/O Multiplexing

Confirm Nginx was compiled with epoll: nginx -V 2>&1 | grep -i event Optional runtime check with strace to see epoll_wait usage.

Step 5: Optimize Backend Connection Reuse

Key upstream configuration (keepalive, least_conn, HTTP/1.1, clear Connection header, enable proxy_socket_keepalive).

Step 6: Enable HTTP/2 and Dynamic Compression

If Nginx lacks http_v2_module, re‑compile with --with-http_v2_module and install.

HTTPS server block with HTTP/2:

server {
    listen 443 ssl http2 reuseport;
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
}

Dynamic gzip settings (already in the http block above).

Step 7: Benchmark and Verify Performance

Install wrk and run a 30‑second test:

wrk -t4 -c100 -d30s --latency http://localhost/health

Sample output shows Requests/sec > 40 k, Avg latency ~2 ms. Adjust parameters until the target > 50 k req/s, P99 < 5 ms.

Step 8: Build Monitoring & Alerting

Prometheus scrape configuration for the Nginx exporter:

scrape_configs:
  - job_name: 'nginx'
    static_configs:
      - targets: ['localhost:9113']

Key metrics to watch:

nginx_requests_total

nginx_connection_accepted_total

nginx_connection_active / waiting

nginx_upstream_response_time_seconds

Sample alert rules for high error rate, high latency, and connection saturation.

Performance Benchmarks & Results

Metric

Before

After

Improvement

Single‑node QPS

50 k

500 k

10×

P99 latency

100 ms

5 ms

20×

Memory usage

2 GB

2.5 GB

1.25×

CPU utilization

85 %

75 %

≈11 % reduction

Best Practices

Parameter isolation : keep kernel, system limits, and Nginx config in separate layers.

Benchmark before and after : always validate changes with load testing.

Monitoring first : establish baseline metrics to compare improvements.

Backend connection reuse : configure keepalive; otherwise performance drops ~50 %.

Asynchronous logging : enable buffered logs to avoid I/O blocking under high load.

CPU affinity : bind workers to specific cores to reduce cache misses.

Regular audits : monthly check of sysctl and ulimit values to ensure they remain effective.

Appendix: Quick Optimization Checklist

# 1. System parameters
cat >> /etc/sysctl.conf <<EOF
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
net.core.netdev_max_backlog = 65535
net.ipv4.tcp_max_tw_buckets = 1000000
net.ipv4.ip_local_port_range = 1024 65535
EOF
sudo sysctl -p

# 2. File descriptor limits
sudo sed -i 's/^#.*soft.*nofile.*/\* soft nofile 100000/' /etc/security/limits.conf
sudo sed -i 's/^#.*hard.*nofile.*/\* hard nofile 100000/' /etc/security/limits.conf

# 3. Verify
sysctl -a | grep somaxconn
ulimit -n

# 4. Nginx config test
sudo nginx -t

# 5. Load test
wrk -t4 -c100 -d30s http://localhost/health

Conclusion : Nginx high‑concurrency tuning is a systematic engineering effort. By combining kernel network tweaks, resource‑limit adjustments, worker‑process optimization, backend keepalive, and real‑time monitoring, throughput can jump from 50 k QPS to 500 k QPS while reducing P99 latency from 100 ms to 5 ms.