Mastering Nginx: Deep Dive into Process Model and Performance Tuning

Background and Purpose

Nginx is the most widely used web server and reverse proxy; many operations engineers have used it for years but only understand superficial configuration. This article starts from the Nginx process model, explains the worker process mechanism, and shows how each directive works to achieve effective optimization.

Prerequisites

Assumes basic Linux skills, knowledge of web servers, and hands‑on experience deploying Nginx.

Environment

Based on Nginx 1.26.x (mainline), Rocky Linux 9.4, OpenSSL 3.2.x.

1. Nginx Process Model

1.1 Overview

Nginx uses a multi‑process architecture instead of threads. The master process handles configuration loading, worker management, signal handling and hot upgrades.

Read and validate configuration files

Manage worker processes (start, stop, reload)

Receive external signals (e.g., nginx -s reload)

Handle hot deployment

The worker processes actually serve client requests.

Accept client connections

Process HTTP requests

Communicate with upstream servers (reverse proxy)

Cache handling

Log writing

When caching is enabled, a cache loader and cache manager exist.

Cache loader – loads cache metadata into memory

Cache manager – manages expiration and cleanup

# 查看Nginx进程结构
ps aux | grep nginx

# 进程关系图
# [master process]
#   |--- worker
#   |--- worker
#   |--- worker
# All workers share the same listening port

1.2 Why Multi‑process, Not Multi‑thread?

Thread‑related issues:

Shared address space makes thread‑safety complex

A single thread crash can bring down the whole process

Memory synchronization overhead is high

Debugging is difficult

Multi‑process advantages:

Workers are isolated; a crash affects only that worker

Each worker can run on a separate CPU core

Memory management is simpler because processes do not share memory

Thundering‑herd problem (all workers wake for a new connection) was fixed in Nginx 1.11.3+ with socket sharding and later with SO_REUSEPORT.

1.3 Connection Handling Model

Nginx uses an event‑driven architecture based on I/O multiplexing (epoll on Linux, kqueue on FreeBSD).

# 查看Nginx事件模型
nginx -V 2>&1 | grep -o 'with-.*'

# 查看系统支持的事件模型
cat /proc/sys/fs/inotify/max_user_watches

Request flow:

Client Request
   |
   v
[Master Process] --listen()--> Port 80/443
   |
   v
[Kernel accepts connection]
   |
   v
[epoll/kqueue notifies ONE worker]
   |
   v
[Worker processes request]
   |-- Static file (sendfile)
   |-- Proxy to backend (uwsgi/fastcgi/http)
   |-- Memcached
   |-- gRPC
   v
Response to Client

2. worker_processes and CPU Core Binding

2.1 worker_processes Directive

worker_processes

controls the number of worker processes.

# 自动检测CPU核心数（推荐配置）
worker_processes auto;

# 固定值
worker_processes 4;

Best practice: use auto in production so Nginx matches the number of CPU cores.

If multiple Nginx instances run on different ports, total workers should not exceed CPU cores.

Bind workers to specific cores with worker_cpu_affinity.

2.2 worker_cpu_affinity Binding

Binding workers to specific cores reduces context‑switch overhead.

# 4‑core CPU binding example
worker_processes 4;
worker_cpu_affinity 0001 0010 0100 1000;

# 8‑core CPU binding example
worker_cpu_affinity 00000001 00000010 00000100 00001000 00010000 00100000 01000000 10000000;

# 8‑core but only 4 workers
worker_processes 4;
worker_cpu_affinity 00010001 00100010 01000100 10001000;

Verify binding:

# 查看进程CPU亲和性
ps -eo pid,psr,comm | grep nginx
# PSR column shows the current CPU core

2.3 worker_priority Adjustment

# Set worker process priority (-20 to 20, lower value = higher priority)
worker_priority -10;

2.4 worker_rlimit_nofile

Sets the maximum number of file descriptors a single worker can open.

# System‑level limits (/etc/security/limits.conf)
# * soft nofile 65535
# * hard nofile 65535

worker_rlimit_nofile 65535;

File‑descriptor consumption sources:

Client connections (one fd per connection)

Upstream connection pool

Opened cache files

Log files

# 查看系统级文件描述符限制
cat /proc/sys/fs/file-max

# 查看当前使用量
cat /proc/sys/fs/file-nr

3. worker_connections and Concurrency Control

3.1 worker_connections Details

Maximum concurrent connections each worker can handle.

events {
    # default 512, increase for production
    worker_connections 1024;
    use epoll;
    # aio on;   # enable async I/O
    multi_accept on;   # accept multiple connections per accept()
}

Theoretical maximum concurrency = worker_processes * worker_connections, but actual capacity must consider fd usage, browser per‑host limits, and upstream connections.

# Example:
# worker_processes = 4
# worker_connections = 1024
# Max concurrency = 4 * 1024 = 4096
# Reserve half for upstream → effective client connections ≈ 2048

3.2 Max Connection Calculation

# nginx.conf
worker_processes auto;
worker_connections 4096;

# Max TCP connections = worker_processes * worker_connections
# Max concurrent clients ≈ Max TCP connections / 2 (HTTP Keep‑Alive)

3.3 Practical Configuration Examples

# General web server
worker_processes auto;
worker_connections 4096;
worker_rlimit_nofile 65535;

events {
    use epoll;
    multi_accept on;
    worker_connections 4096;
}

# High‑concurrency static file server
worker_processes auto;
worker_connections 16384;
worker_rlimit_nofile 262144;

events {
    use epoll;
    multi_accept on;
    worker_connections 16384;
}

# Reverse‑proxy server (reserve upstream connections)
worker_processes auto;
worker_connections 8192;
worker_rlimit_nofile 32768;

events {
    use epoll;
    multi_accept on;
}

4. Upstream Load‑Balancing Strategies

4.1 round_robin (default)

upstream backend {
    server 192.168.1.101:8080;
    server 192.168.1.102:8080;
    server 192.168.1.103:8080;
}
server {
    location / {
        proxy_pass http://backend;
    }
}

4.2 Weighted round‑robin

# Equal weight
upstream backend {
    server 192.168.1.101:8080 weight=5;
    server 192.168.1.102:8080 weight=5;
    server 192.168.1.103:8080 weight=5;
}

# Unequal weight
upstream backend {
    server 192.168.1.101:8080 weight=3;
    server 192.168.1.102:8080 weight=2;
    server 192.168.1.103:8080 weight=1;
}

4.3 ip_hash

upstream backend {
    ip_hash;
    server 192.168.1.101:8080;
    server 192.168.1.102:8080;
    server 192.168.1.103:8080;
}

4.4 least_conn

upstream backend {
    least_conn;
    server 192.168.1.101:8080;
    server 192.168.1.102:8080;
    server 192.168.1.103:8080;
}

4.5 least_time (Nginx Plus only)

# Nginx Plus example
upstream backend {
    least_time header;
    server 192.168.1.101:8080;
    server 192.168.1.102:8080;
}

4.6 Full upstream example with backup and keepalive

upstream tomcat_cluster {
    least_conn;
    server 192.168.1.101:8080 weight=3 max_fails=3 fail_timeout=30s;
    server 192.168.1.102:8080 weight=2 max_fails=3 fail_timeout=30s;
    server 192.168.1.103:8080 backup;  # backup server

    keepalive 32;
}
# Parameter notes:
# weight=N – weighted round‑robin weight
# max_fails=N – max failures before marking down
# fail_timeout – time to consider a server down
# backup – designate as backup
# down – permanently unavailable

5. Proxy Buffering and Caching

5.1 Buffering Basics

Client <--> Nginx Worker <--> Upstream Server
   |-- proxy_buffer_size: reads response headers
   |-- proxy_buffers: reads response body
   |-- proxy_busy_buffers_size: buffers while sending to client
   |-- proxy_cache: cache storage

5.2 Basic Proxy Buffer Settings

location / {
    proxy_pass http://backend;

    proxy_buffer_size 4k;
    proxy_buffers 8 16k;
    proxy_busy_buffers_size 32k;

    proxy_connect_timeout 60s;
    proxy_read_timeout 60s;
    proxy_send_timeout 60s;
}

5.3 High‑Concurrency Proxy

location /api/ {
    proxy_pass http://backend;

    proxy_buffer_size 64k;
    proxy_buffers 16 64k;
    proxy_buffering on;

    proxy_busy_buffers_size 128k;
    proxy_download_rate 0;   # unlimited
}

5.4 Disable Buffering (real‑time streaming)

location /streaming/ {
    proxy_pass http://backend;
    proxy_buffering off;
    proxy_cache off;

    proxy_http_version 1.1;
    proxy_set_header Connection "upgrade";
}

5.5 HTTP Cache Configuration

# Define cache zone
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m use_temp_path=off;

location / {
    proxy_pass http://backend;
    proxy_cache my_cache;

    proxy_cache_key "$scheme$request_method$host$request_uri";

    proxy_cache_valid 200 60m;
    proxy_cache_valid 404 1m;

    add_header X-Cache-Status $upstream_cache_status;
    proxy_cache_bypass $cookie_nocache $arg_nocache;
}

Cache‑status values: MISS, HIT, EXPIRED, STALE, UPDATING, REVALIDATED.

6. Keepalive Connections

6.1 Upstream keepalive

upstream backend {
    server 192.168.1.101:8080;
    server 192.168.1.102:8080;

    keepalive 32;
}
server {
    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

6.2 Client keepalive

server {
    listen 80;
    server_name example.com;

    keepalive_timeout 65;
    keepalive_requests 1000;

    tcp_nopush off;
    tcp_nodelay on;
}

7. Gzip and Brotli Compression

7.1 Gzip Basics

http {
    gzip on;
    gzip_min_length 1024;
    gzip_comp_level 4;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml;
    gzip_vary on;
    gzip_proxied any;
    gzip_disable "msie6";
}

7.2 Detailed Gzip Parameters

http {
    gzip on;
    gzip_min_length 256;
    gzip_comp_level 4;
    gzip_types text/plain text/css text/xml text/javascript application/json application/javascript application/xml application/xml+rss application/x-javascript image/svg+xml;
    gzip_vary on;
    gzip_proxied any;
    gzip_disable "MSIE [1-6]\\.";
}

7.3 Brotli (requires module)

http {
    brotli on;
    brotli_types text/plain text/css application/json application/javascript text/xml application/xml image/svg+xml;
    brotli_comp_level 4;
    brotli_min_length 256;
}

7.4 Compression Comparison

# Test gzip
curl -I -H "Accept-Encoding: gzip" http://example.com/api/data
# Result shows Content-Encoding: gzip

# Size comparison (JSON example)
# Original: 50KB
# Gzip: 12KB (76% reduction)
# Brotli: 10KB (80% reduction)

8. Static Resource Caching

8.1 Cache Policy

server {
    listen 80;
    server_name static.example.com;
    root /var/www/static;

    # CSS/JS – 1 year
    location ~* \.(css|js)$ {
        expires 1y;
        add_header Cache-Control "public, immutable";
        access_log off;
    }

    # Images – 30 days
    location ~* \.(jpg|jpeg|png|gif|ico|webp|svg)$ {
        expires 30d;
        add_header Cache-Control "public";
        access_log off;
    }

    # Fonts – 1 year
    location ~* \.(woff|woff2|ttf|eot)$ {
        expires 1y;
        add_header Cache-Control "public";
        access_log off;
    }

    # HTML – no cache
    location ~* \.html$ {
        expires -1;
        add_header Cache-Control "no-cache, no-store";
    }
}

8.2 Static Performance Optimizations

http {
    open_file_cache max=1000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    client_body_timeout 12;
    client_header_timeout 12;
}

8.3 CORS Configuration

location /assets/ {
    add_header 'Access-Control-Allow-Origin' '*';
    add_header 'Access-Control-Allow-Methods' 'GET, OPTIONS';
    add_header 'Access-Control-Max-Age' 86400;

    if ($request_method = 'OPTIONS') {
        add_header 'Access-Control-Allow-Origin' '*';
        add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
        add_header 'Access-Control-Max-Age' 86400;
        add_header 'Content-Type' 'text/plain charset=UTF-8';
        add_header 'Content-Length' 0;
        return 204;
    }

    expires 30d;
    add_header Cache-Control "public";
}

9. SSL/TLS Handshake Optimizations

9.1 Basic HTTPS

server {
    listen 443 ssl http2;
    server_name example.com;

    ssl_certificate /etc/nginx/ssl/example.com.crt;
    ssl_certificate_key /etc/nginx/ssl/example.com.key;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
    ssl_prefer_server_ciphers off;

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;

    ssl_stapling on;
    ssl_stapling_verify on;
}

9.2 Modern SSL (TLS 1.3 only)

server {
    listen 443 ssl http2;
    server_name example.com;

    ssl_certificate /etc/nginx/ssl/fullchain.pem;
    ssl_certificate_key /etc/nginx/ssl/privkey.pem;

    ssl_protocols TLSv1.3;
    ssl_ciphers TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    add_header Strict-Transport-Security "max-age=63072000" always;
    ssl_trusted_certificate /etc/nginx/ssl/chain.pem;
}

9.3 SSL Performance

http {
    ssl_session_cache shared:SSL:50m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;

    ssl_stapling on;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;
}
server {
    listen 443 ssl http2;
    server_name example.com;

    ssl_certificate /etc/nginx/ssl/example.com.crt;
    ssl_certificate_key /etc/nginx/ssl/example.com.key;

    ssl_early_data on;   # 0‑RTT
    keepalive_timeout 100;
}

9.4 HTTP/2 Tweaks

server {
    listen 443 ssl http2;
    server_name example.com;

    # http2_push_preload on;   # optional, superseded by HTTP/3
    http2_max_concurrent_streams 128;
    http2_recv_buffer_size 256k;
}

10. Core Configuration Checklist

worker_processes : auto – automatically matches CPU cores.

worker_connections : 65535 – maximum concurrent connections per worker.

worker_cpu_affinity : auto – CPU core binding.

multi_accept : on – accept multiple connections per accept call.

sendfile : on – efficient file transfer.

tcp_nopush : on – optimizes sendfile.

tcp_nodelay : on – disables Nagle algorithm.

keepalive_timeout : 65 – client keep‑alive duration.

gzip : on – enable compression.

ssl_protocols : TLSv1.2 TLSv1.3 – disable old protocols.

11. Configuration Validation and Load Testing

11.1 Syntax Check

# Test configuration syntax
nginx -t
# Show full configuration
nginx -T
# Test specific file
nginx -t -c /path/to/nginx.conf
# Show compile‑time options
nginx -V

11.2 Load‑testing Tools

# Install wrk
dnf install wrk -y

# Basic test
wrk -t12 -c400 -d30s http://127.0.0.1/

# Lua script example
wrk -t12 -c400 -d30s -s post.lua http://127.0.0.1/api

11.3 ApacheBench (ab)

# Install
dnf install httpd-tools -y

# Simple test
ab -n 100000 -c 1000 http://127.0.0.1/

# POST test
ab -n 10000 -c 100 -p data.json -T application/json http://127.0.0.1/api

11.4 Load‑test Script Example

#!/bin/bash
# nginx_load_test.sh
TARGET_URL="${1:-http://127.0.0.1/}"
REPORT_FILE="/tmp/nginx_load_test_$(date +%Y%m%d_%H%M%S).txt"

echo "=== Nginx Performance Test ==="
echo "Target URL: ${TARGET_URL}"
echo "Start time: $(date)"

echo "
【1】Low concurrency (100, 30s)..."
wrk -t4 -c100 -d30s "${TARGET_URL}" 2>&1 | tee -a "${REPORT_FILE}"

echo "
【2】Medium concurrency (500, 30s)..."
wrk -t8 -c500 -d30s "${TARGET_URL}" 2>&1 | tee -a "${REPORT_FILE}"

echo "
【3】High concurrency (1000, 60s)..."
wrk -t16 -c1000 -d60s "${TARGET_URL}" 2>&1 | tee -a "${REPORT_FILE}"

echo "
【4】Slow‑connection test (100, latency)..."
wrk -t4 -c100 -d30s --latency "${TARGET_URL}" 2>&1 | tee -a "${REPORT_FILE}"

echo "
Test completed, report saved to: ${REPORT_FILE}"

11.5 Troubleshooting Scripts

#!/bin/bash
# nginx_troubleshoot.sh
echo "=== Nginx Troubleshooting ==="

echo "
【1】Nginx process status:"
systemctl status nginx | head -10

echo "
【2】Configuration syntax:"
nginx -t 2>&1

echo "
【3】Port usage:"
ss -tlnp | grep -E ':80|:443'

echo "
【4】Recent error log:"
tail -20 /var/log/nginx/error.log

echo "
【5】Backend check:"
curl -s -o /dev/null -w "HTTP %{http_code}" http://127.0.0.1:8080/health || echo "Backend unreachable"

echo "
【6】Cache directory:"
ls -la /var/cache/nginx/ 2>/dev/null || echo "No cache dir"

echo "
【7】Disk space:"
df -h /var/log/nginx

11.6 Monitoring Script Example

#!/bin/bash
# nginx_monitor.sh
ALERT_EMAIL="[email protected]"

check_nginx() {
    if ! systemctl is-active nginx &>/dev/null; then
        echo "CRITICAL: Nginx is not running!"
        return 1
    fi
    worker_count=$(ps aux | grep "nginx: worker" | grep -v grep | wc -l)
    if [ "$worker_count" -eq 0 ]; then
        echo "CRITICAL: No worker processes!"
        return 1
    fi
    echo "OK: Nginx running with $worker_count workers"
    return 0
}

check_connections() {
    active=$(curl -s http://127.0.0.1/nginx_status | grep "Active" | awk '{print $2}')
    reading=$(curl -s http://127.0.0.1/nginx_status | grep "Reading" | awk '{print $2}')
    writing=$(curl -s http://127.0.0.1/nginx_status | grep "Writing" | awk '{print $2}')
    waiting=$(curl -s http://127.0.0.1/nginx_status | grep "Waiting" | awk '{print $2}')
    echo "Connections: active=$active reading=$reading writing=$writing waiting=$waiting"
}

check_nginx
check_connections

12. Common Configuration Errors and Fixes

12.1 Typical Errors

403 Forbidden

# Check file permissions
ls -la /var/www/html/index.html
# Fix ownership and mode
chown nginx:nginx /var/www/html/index.html
chmod 644 /var/www/html
# SELinux context
ls -Z /var/www/html/index.html
chcon -t httpd_sys_content_t /var/www/html/index.html

502 Bad Gateway

# Verify upstream is running
curl -v http://127.0.0.1:8080/health

# Check nginx error log for upstream messages
tail -50 /var/log/nginx/error.log | grep "upstream"

# Adjust timeouts
proxy_connect_timeout 60s;
proxy_read_timeout 60s;

504 Gateway Timeout

# Increase timeouts
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;

# Or investigate backend performance
upstream backend {
    least_conn;
    server 127.0.0.1:8080 weight=3;
}

12.2 Diagnostic Scripts

#!/bin/bash
# nginx_monitor.sh
ALERT_EMAIL="[email protected]"

check_nginx() {
    if ! systemctl is-active nginx &>/dev/null; then
        echo "CRITICAL: Nginx is not running!"
        return 1
    fi
    worker_count=$(ps aux | grep "nginx: worker" | grep -v grep | wc -l)
    if [ "$worker_count" -eq 0 ]; then
        echo "CRITICAL: No worker processes!"
        return 1
    fi
    echo "OK: Nginx running with $worker_count workers"
    return 0
}

check_connections() {
    active=$(curl -s http://127.0.0.1/nginx_status | grep "Active" | awk '{print $2}')
    reading=$(curl -s http://127.0.0.1/nginx_status | grep "Reading" | awk '{print $2}')
    writing=$(curl -s http://127.0.0.1/nginx_status | grep "Writing" | awk '{print $2}')
    waiting=$(curl -s http://127.0.0.1/nginx_status | grep "Waiting" | awk '{print $2}')
    echo "Connections: active=$active reading=$reading writing=$writing waiting=$waiting"
}

check_nginx
check_connections

13. References

Nginx official documentation: https://nginx.org/en/docs/

Nginx Performance Tuning blog: https://www.nginx.com/blog/tuning-nginx/

《高性能Linux服务器构建实战》