Master Linux Network Tuning for High‑Concurrency: A Practical Guide

Introduction

In high‑concurrency network services, Linux kernel defaults often become bottlenecks, causing performance degradation, connection timeouts, or crashes. This guide analyzes a real case, explains parameter meanings, diagnoses problems, and offers hands‑on tuning to support millions of concurrent connections.

1. Problem Background

1.1 Test Environment

Server configuration: vCPU 8 cores, 16 GB RAM, 4 Gbps bandwidth, 800 kpps.

1.2 Initial Parameter Analysis

Using sysctl the original settings are:

net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 131072
net.ipv4.ip_local_port_range = 1024 61999

The main defects are a small half‑open queue, narrow port range, and strict buffer limits.

2. Deep Diagnosis

2.1 Connection‑State Monitoring

Real‑time TCP state statistics:

watch -n 1 'netstat -ant | awk "/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

Sample output shows many TIME_WAIT and SYN_RECV connections.

2.2 Key Parameter Interpretation

tcp_max_syn_backlog

: half‑open queue length, default 8192, may overflow under burst traffic. somaxconn: full‑connection queue length, must match application backlog. tcp_tw_reuse: enables fast reuse of TIME_WAIT ports, disabled by default. tcp_rmem / tcp_wmem: read/write buffer sizes, maximum only 6 MB, limiting throughput.

3. Tuning Solutions

3.1 Connection Management

Resolve TIME_WAIT accumulation:

echo "net.ipv4.tcp_tw_reuse = 1" >> /etc/sysctl.conf
echo "net.ipv4.tcp_max_tw_buckets = 262144" >> /etc/sysctl.conf
echo "net.ipv4.ip_local_port_range = 1024 65000" >> /etc/sysctl.conf

Shorten connection recycle time:

echo "net.ipv4.tcp_fin_timeout = 30" >> /etc/sysctl.conf

3.2 Queue and Buffer Optimization

Expand connection queues:

echo "net.ipv4.tcp_max_syn_backlog = 65535" >> /etc/sysctl.conf
echo "net.core.somaxconn = 65535" >> /etc/sysctl.conf
echo "net.core.netdev_max_backlog = 10000" >> /etc/sysctl.conf

Adjust memory buffers:

cat >> /etc/sysctl.conf <<EOF
net.ipv4.tcp_mem = 8388608 12582912 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
EOF

3.3 Keepalive and Timeout

echo "net.ipv4.tcp_keepalive_time = 600" >> /etc/sysctl.conf
echo "net.ipv4.tcp_keepalive_intvl = 30" >> /etc/sysctl.conf

4. Validation and Monitoring

4.1 Real‑time Monitoring Script

#!/bin/bash
while true; do
  clear
  date
  echo "---- TCP状态 ----"
  netstat -ant | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
  echo "---- 半连接队列 ----"
  ss -ltn | awk 'NR>1 {print "Listen队列: Recv-Q="$2", Send-Q="$3}'
  echo "---- 端口使用率 ----"
  echo "已用端口: $(netstat -ant | grep -v LISTEN | awk '{print $4}' | cut -d: -f2 | sort -u | wc -l)/$((65000-1024))"
  sleep 5
done

4.2 Prometheus Alert Example

alert: TCP_SYN_Dropped
expr: increase(node_netstat_Tcp_Ext_SyncookiesFailed{job="node"}[1m]) > 0
for: 5m
labels:
  severity: critical
annotations:
  summary: "SYN队列溢出 (实例 {{ $labels.instance }})"

4.3 Load‑Testing Recommendation

Use wrk to simulate high load, e.g. wrk -t16 -c10000 -d60s http://service:8080, and monitor SYN_RECV, netstat packet loss, and memory usage.

5. Pitfalls

5.1 Common Misconceptions

Avoid blindly enabling tcp_tw_recycle; it breaks NAT and has been removed since Linux 4.12.

Too large buffers can cause OOM; tune tcp_mem according to available RAM.

5.2 Parameter Dependencies

somaxconn

must be greater than or equal to the application’s backlog, e.g., Nginx listen 80 backlog=65535;.

6. Conclusion

After applying the above tuning, TIME_WAIT connections dropped by roughly 70 %, maximum concurrent connections exceeded 30 k, and network throughput doubled.