Skip to content
Kernel Parameter Tuning for Nginx: A Verified Methodology for 300% Concurrency Gains
Operations 25 min read

Kernel Parameter Tuning for Nginx: A Verified Methodology for 300% Concurrency Gains

This article presents a systematic, hypothesis-driven methodology for tuning Linux kernel parameters to improve Nginx concurrency, covering file descriptor limits, listen queue configuration, network buffer tuning, and safe canary deployment with rollback procedures, all validated through metrics and load testing.

Raymond Ops
Raymond Ops
Raymond Ops

1. Turn "Concurrency Insufficiency" into a Verifiable Hypothesis

Before tuning, define success thresholds under the same request model: sustained QPS, p95/p99 latency, business 5xx, upstream 5xx, ListenOverflows increment, CPU softirq, NIC drops, and memory watermarks. A peak QPS rise with doubled tail latency or upstream connection refusals is not success.

Record version, compile options, effective config, and service limits. Nginx worker config may come from include files; do not rely solely on the main config.

nginx -V 2>&1
nginx -T 2>&1 | sed -n '1,240p'
systemctl show nginx -p LimitNOFILE -p ExecStart --no-pager

Check master and worker file descriptor usage. /proc shows instantaneous state; collection must cover the load-test window or real peak window.

MASTER_PID="$(systemctl show -p MainPID --value nginx)"
printf 'master pid: %s
' "${MASTER_PID}"
ps -o pid,ppid,comm,nlwp,etime -C nginx
for pid in $(pgrep -x nginx); do
  printf 'pid=%s open_fds=' "${pid}"
  find "/proc/${pid}/fd" -maxdepth 1 -type l | wc -l
done
cat /proc/sys/fs/file-nr
file-nr

alone cannot prove exhaustion. Cross-validate with Nginx log EMFILE/ENFILE or accept4 failures and worker Max open files limit.

START_TIME="<开始时间,例如 2026-07-24 09:00:00>"
journalctl -u nginx --since "${START_TIME}" --no-pager \
  | rg -i 'worker_connections|too many open files|accept4\(|EMFILE|ENFILE'

Read listen socket, connection states, and queues. For listening TCP sockets, Recv-Q is completed handshakes not yet taken by the application; sustained approach to Send-Q indicates accept path or backlog as a bottleneck candidate.

ss -ltnp '( sport = :80 or sport = :443 )'
ss -s
ss -tan state established '( sport = :80 or sport = :443 )' | wc -l
ss -tan state time-wait | wc -l

If observing Nginx connection states, configure stub_status but expose only to localhost or controlled monitoring segment, not public internet.

server {
  listen 127.0.0.1:8080;
  server_name localhost;
  location = /nginx_status {
    stub_status;
    allow 127.0.0.1;
    deny all;
  }
}
curl --fail --silent --show-error http://127.0.0.1:8080/nginx_status

Kernel counters are the evidence for actual listen queue overflow. Continuous sampling showing growth in ListenOverflows or ListenDrops warrants considering queue increase or faster application accept; Syncookies growth requires investigating burst connection storms and abnormal SYN.

nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPBacklogDrop \
  TcpExtSyncookiesSent TcpExtSyncookiesRecv
sleep 60
nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPBacklogDrop \
  TcpExtSyncookiesSent TcpExtSyncookiesRecv

Check CPU, softirq, and driver stats. If a specific CPU's NET_RX is especially high and NIC drops grow simultaneously, merely adding worker_connections will not solve the packet reception bottleneck.

mpstat -P ALL 1 5
cat /proc/softirqs | sed -n '1,14p'
ip -s link show dev <网卡名>
ethtool -S <网卡名> 2>/dev/null | rg -i 'drop|miss|err|timeout' || true

Save isomorphic snapshots at each stage to later determine what changes actually affected. The following script only collects data, does not modify the system.

#!/usr/bin/env bash
set -euo pipefail
OUTPUT_DIR="<快照目录>"
STAMP="$(date +%Y%m%d_%H%M%S)"
mkdir -p "${OUTPUT_DIR}"
{
  date --iso-8601=seconds
  uname -a
  nginx -v 2>&1
  sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog fs.file-max
  ss -s
  nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPBacklogDrop
} > "${OUTPUT_DIR}/nginx_kernel_${STAMP}.txt"

Load testing creates real load; run only in isolated environments, staging, or approved production canary ingress. Request path must be idempotent; do not use write APIs.

wrk -t4 -c400 -d120s --latency \
  -H 'Host: <业务域名>' \
  https://<入口IP或域名>/<只读请求路径>

2. Handle Process and File Descriptor Limits First

A single client connection in reverse proxy may simultaneously consume downstream socket, upstream socket, log file, and temporary file. Thus worker_processes × worker_connections is a rough boundary, not equal to usable client concurrency. Long connections, WebSocket, caching, and upstream keepalive change the actual model.

Extract worker and connection model, and confirm actual visible CPU count in container. worker_processes auto is affected by CPU quota; do not assume host core count.

nginx -T 2>&1 | rg -n \
  '^worker_processes|^worker_rlimit_nofile|worker_connections|multi_accept|use epoll|keepalive|reuseport'
nproc
getconf OPEN_MAX

Check for existing systemd drop-in. Do not edit vendor files under /usr/lib/systemd/system directly; upgrades may overwrite changes.

systemctl cat nginx
systemctl show nginx -p FragmentPath -p DropInPaths -p LimitNOFILE --no-pager

Create a dedicated drop-in. <打开文件上限> must cover peak sockets, upstream connections, log files, and headroom; fs.file-max must cover total across all services on the machine. sudo systemctl edit nginx In the editor write:

[Service]
LimitNOFILE=<打开文件上限>

Backup source config and expanded effective config before changes. Backup directory should be permission-protected and recorded in the change ticket.

#!/usr/bin/env bash
set -euo pipefail
NGINX_CONF="<nginx配置文件>"
BACKUP_DIR="/var/backups/nginx-$(date +%Y%m%d_%H%M%S)"
sudo install -d -m 0700 "${BACKUP_DIR}"
sudo cp -a "${NGINX_CONF}" "${BACKUP_DIR}/"
sudo nginx -T > "${BACKUP_DIR}/nginx-effective.conf" 2>&1
printf 'backup directory: %s
' "${BACKUP_DIR}"

Explicitly set worker strategy in main and events blocks. On Linux epoll is usually auto-selected; explicit declaration aids audit. multi_accept on suits burst connections but must be observed alongside worker distribution, tail latency, and upstream pressure.

user nginx;
worker_processes auto;
worker_rlimit_nofile <打开文件上限>;

events {
  worker_connections <单worker连接上限>;
  use epoll;
  multi_accept on;
}
LimitNOFILE

is inherited only by new workers. Restart rebuilds workers and may affect long connections; must drain traffic or be in approved maintenance window; ordinary config changes can use reload.

sudo nginx -t
sudo systemctl daemon-reload
sudo systemctl restart nginx
systemctl show nginx -p LimitNOFILE --no-pager

Actual process limit is per /proc. If soft limit is less than worker_rlimit_nofile, connection config will not take effect as expected.

for pid in $(pgrep -x nginx); do
  echo "=== pid ${pid} ==="
  awk '/Max open files/ {print}' "/proc/${pid}/limits"
done

3. Listen Queue and TCP Connection Establishment: Don't Conflate Two Queues

listen backlog

is truncated by net.core.somaxconn; setting a large Nginx backlog while somaxconn is small does not enlarge the completed connection queue. The SYN half-connection queue is constrained by tcp_max_syn_backlog, syncookies, and connection rate; it is not the same object as the accept queue.

Read current values, record originals, and confirm parameters exist. If missing, check kernel and distro docs; do not arbitrarily replace with similar fields.

sysctl net.core.somaxconn
sysctl net.ipv4.tcp_max_syn_backlog
sysctl net.ipv4.tcp_syncookies
sysctl net.ipv4.tcp_synack_retries

Set explicit backlog on listen ports when needed. reuseport changes multi-worker connection distribution and affects long connections, L4 forwarding, and health checks; must canary on a single node first.

server {
  listen 80 backlog=<监听队列长度> reuseport;
  listen 443 ssl http2 backlog=<监听队列长度> reuseport;
  server_name <业务域名>;
  location / {
    proxy_pass http://app_backend;
  }
}

Manage changes via a dedicated sysctl file to avoid mixing with historical config. tcp_syncookies=1 should be kept as protection; do not disable to chase a counter value.

# /etc/sysctl.d/60-nginx-concurrency.conf
net.core.somaxconn = <监听队列长度>
net.ipv4.tcp_max_syn_backlog = <SYN队列长度>
net.ipv4.tcp_syncookies = 1
fs.file-max = <系统文件句柄上限>

Load a single file to immediately modify running kernel; execute on a drainable node first. Do not run sysctl --system directly because it loads historical files from multiple directories, expanding blast radius.

sudo sysctl --load=/etc/sysctl.d/60-nginx-concurrency.conf
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog \
  net.ipv4.tcp_syncookies fs.file-max

Verify listen port Send-Q with ss, and combine with nstat and load test to determine if overflow persists. A single empty queue does not mean no problem at peak. ss -ltn '( sport = :80 or sport = :443 )' High TIME_WAIT does not automatically mean kernel parameters need tuning. A more common optimization for Nginx-to-upstream short connections is upstream keepalive; it must be validated with upstream max connections, failover, and protocol support.

upstream app_backend {
  server <上游IP1>:<端口> max_fails=3 fail_timeout=10s;
  server <上游IP2>:<端口> max_fails=3 fail_timeout=10s;
  keepalive <上游空闲连接数>;
}

server {
  location / {
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_pass http://app_backend;
  }
}

Observe upstream connection states and error logs. Root cause conclusion requires at least two categories of evidence from connection states, Nginx logs, and backend metrics.

ss -tan dst <上游IP1>:<端口> | awk 'NR==1 || $1 ~ /ESTAB|TIME-WAIT/'
journalctl -u nginx --since '10 min ago' --no-pager \
  | rg -i 'upstream.*(timed out|prematurely closed|connect\( failed)'

4. Packet Reception Path: Tune Only When Metrics Support It

Only enter NIC tuning when softirq backlog, driver drop counters, or softnet drops grow in the same window. /proc/net/softnet_stat is per-CPU hexadecimal counts; cannot judge by eye.

sar -n DEV 1 10
cat /proc/net/softnet_stat
ethtool -g <网卡名>
ethtool -l <网卡名>

Inspect RSS, RPS, and receive queues. Cloud NICs, VMs, and CNI may not support all ethtool features; on failure, record capability boundaries rather than silently ignoring in scripts.

ethtool -x <网卡名> 2>/dev/null || true
for queue in /sys/class/net/<网卡名>/queues/rx-*; do
  printf '%s: ' "${queue}"
  cat "${queue}/rps_cpus"
done

Modifying combined queue may cause brief interface reset and remote connection interruption. Must have out-of-band console, node drain, and rollback channel before execution; <队列数> must not exceed maximum shown by ethtool -l.

sudo ethtool -L <网卡名> combined <队列数>
sudo ethtool -l <网卡名>

If packet reception backlog is confirmed, then evaluate backlog and socket buffers. Larger buffers increase memory usage under high concurrency; must set rollback thresholds and canary on a single node.

# /etc/sysctl.d/61-nginx-network-buffer.conf
net.core.netdev_max_backlog = <网卡收包积压上限>
net.core.rmem_max = <最大接收缓冲字节数>
net.core.wmem_max = <最大发送缓冲字节数>

After applying, immediately compare memory, socket summary, and NIC counters. Changes that increase memory without reducing drops or tail latency should be rolled back immediately.

sudo sysctl --load=/etc/sysctl.d/61-nginx-network-buffer.conf
free -h
ss -s
ss -m state established '( sport = :443 )' | sed -n '1,80p'
ip -s link show dev <网卡名>

5. Canary Validation, Alerting, and Rollback

Change one variable at a time: first file descriptors, then listen queue, then network buffers. If a load balancer is in front, drain or weight-down a single node first; specific commands depend on the product; do not apply fabricated "generic drain commands".

Health checks must include syntax, port, idempotent business path, and key error logs. <健康检查URL> must not cause writes, orders, or cache pollution.

#!/usr/bin/env bash
set -euo pipefail
HEALTH_URL="<健康检查URL>"
sudo nginx -t
curl --fail --silent --show-error --max-time 3 "${HEALTH_URL}" >/dev/null
ERRORS="$(journalctl -u nginx --since '5 min ago' --no-pager \
  | rg -c 'worker_connections are not enough|too many open files|accept4\( failed' || true)"
test "${ERRORS}" -eq 0
printf 'health check passed; critical lines=%s
' "${ERRORS}"

If using Prometheus, metric names follow the actual exporter. Confirm labels and value ranges in the expression browser before setting alerts.

sum(rate(nginx_http_requests_total{job="<nginx任务名>",status=~"5.."}[5m]))
/
sum(rate(nginx_http_requests_total{job="<nginx任务名>"}[5m]))

TIME_WAIT should be an auxiliary observation of connection lifecycle, not a standalone alerting incident metric. Metric names follow actual exporter.

max_over_time(node_sockstat_TCP_tw{instance="<节点>"}[15m])

Listen overflow is more reliably judged by increment. Restarts and exporter resets affect absolute values; this rule also follows actual exporter metrics.

sum by (instance) (
  increase(node_netstat_TcpExt_ListenOverflows[5m])
) > 0

Comparing before/after snapshots only reveals config and counter differences; final judgment must use latency, success rate, upstream load, and error rate from the same load-test window.

#!/usr/bin/env bash
set -euo pipefail
BEFORE_FILE="<变更前快照文件>"
AFTER_FILE="<变更后快照文件>"
diff -u "${BEFORE_FILE}" "${AFTER_FILE}" || true
printf '%s
' '请结合同一压测窗口的延迟、错误率与 nstat 增量人工判定。'

On connection failures, p99 anomalies, rapid memory rise, or upstream pressure, first stop expanding canary and drain the changed node, then collect state. Do not continuously reload, restart, and modify multiple configs during peak.

Rollback Nginx config by restoring exact pre-change backup, syntax check, reload, then verify business health. <备份目录> must be the correct timestamp from the backup script.

sudo cp -a <备份目录>/nginx.conf <nginx配置文件>
sudo nginx -t
sudo systemctl reload nginx
curl --fail --silent --show-error <健康检查URL> >/dev/null

If a new systemd drop-in was created, confirm it carries no other settings, then delete and restart to make new limits effective. This rebuilds workers; execute only in drained or maintenance window.

sudo rm /etc/systemd/system/nginx.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart nginx
systemctl show nginx -p LimitNOFILE --no-pager

Sysctl rollback must not guess defaults. Restore original values from pre-change snapshot or config management; deleting the new file and running sysctl --system may load other historical files, so explicitly restore confirmed original values.

sudo rm /etc/sysctl.d/60-nginx-concurrency.conf
sudo sysctl -w net.core.somaxconn=<变更前somaxconn>
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=<变更前SYN队列>
sudo sysctl -w fs.file-max=<变更前文件句柄上限>

After rollback, collect connection, kernel, NIC, and log evidence again. If anomalies persist, root cause is not in this batch of parameters; return to baseline phase rather than stacking more "high concurrency" configs.

ss -s
nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPBacklogDrop
ip -s link show dev <网卡名>
journalctl -u nginx --since '10 min ago' --no-pager | tail -n 120

Reusable tuning outcomes are not a mysterious parameter table but a traceable causal chain: clear hypothesis, sufficient evidence, one variable changed at a time, same-load validation of gains and costs, safe rollback on failure. Only parameters verified through this chain deserve inclusion in configuration management.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

OperationsConcurrencyperformance testingnginxLinux kernelsysctlrollbackcanary deploymentkernel-tuningfile-descriptorslisten-queuenetwork-buffer
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.