Master Linux Network Tuning for High-Concurrency: Practical Guide

Linux High-Concurrency Network Parameter Tuning Guide

Introduction

In high‑concurrency network services, default Linux kernel network parameters often become bottlenecks, causing performance degradation, connection timeouts, or crashes. This article analyses a real‑world case, explains key parameters, diagnoses issues, and provides step‑by‑step tuning practices to support millions of concurrent connections.

1. Problem Background

1.1 Case Environment

Server configuration: 8 vCPU, 16 GB RAM, 4 Gbps bandwidth, 800 kpps.

Observed anomalies: TIME_WAIT connections accumulated (2464). CLOSE_WAIT connections (4).

Occasional new‑connection timeout.

1.2 Initial Parameter Analysis

Using sysctl the original settings were:

net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 131072
net.ipv4.ip_local_port_range = 1024 61999

Key defects: small half‑open queue, narrow port range, strict buffer limits.

2. Deep Diagnosis

2.1 Connection‑State Monitoring

Real‑time TCP state statistics:

watch -n 1 'netstat -ant | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

Sample output:

ESTABLISHED 790
TIME_WAIT 2464
SYN_RECV 32  # half‑open connections to watch

2.2 Half‑Open Queue Check

# Show SYN_RECV details
ss -ntp state syn-recv
# Monitor listen drops
netstat -s | grep -i 'listen drops'

2.3 Key Parameter Interpretation

Important kernel parameters: tcp_max_syn_backlog: half‑open queue length (default 8192, may overflow under burst traffic). somaxconn: full‑connection queue length (must match application backlog). tcp_tw_reuse: enables rapid reuse of TIME_WAIT ports (disabled by default). tcp_rmem / tcp_wmem: read/write buffer sizes (default max 6 MB, limits throughput).

3. Tuning Solutions

3.1 Connection Management

Resolve TIME_WAIT accumulation:

echo "net.ipv4.tcp_tw_reuse = 1" >> /etc/sysctl.conf
echo "net.ipv4.tcp_max_tw_buckets = 262144" >> /etc/sysctl.conf
echo "net.ipv4.ip_local_port_range = 1024 65000" >> /etc/sysctl.conf

Shorten connection recycle time:

echo "net.ipv4.tcp_fin_timeout = 30" >> /etc/sysctl.conf

3.2 Queue and Buffer Optimization

Expand connection queues:

echo "net.ipv4.tcp_max_syn_backlog = 65535" >> /etc/sysctl.conf
echo "net.core.somaxconn = 65535" >> /etc/sysctl.conf
echo "net.core.netdev_max_backlog = 10000" >> /etc/sysctl.conf

Adjust memory buffers:

cat >> /etc/sysctl.conf <<EOF
net.ipv4.tcp_mem = 8388608 12582912 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
EOF

3.3 Keepalive and Timeout

echo "net.ipv4.tcp_keepalive_time = 600" >> /etc/sysctl.conf
echo "net.ipv4.tcp_keepalive_intvl = 30" >> /etc/sysctl.conf

4. Validation and Monitoring

4.1 Real‑Time Monitoring Script

#!/bin/bash
while true; do
  clear
  date
  echo "---- TCP STATE ----"
  netstat -ant | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
  echo "---- HALF‑OPEN QUEUE ----"
  ss -ltn | awk 'NR>1 {print "Listen queue: Recv-Q="$2", Send-Q="$3}'
  echo "---- PORT USAGE ----"
  echo "Used ports: $(netstat -ant | grep -v LISTEN | awk '{print $4}' | cut -d: -f2 | sort -u | wc -l)/$((65000-1024))"
  sleep 5
done

4.2 Prometheus Alert Example

alert: TCP_SYN_Dropped
expr: increase(node_netstat_Tcp_Ext_SyncookiesFailed{job="node"}[1m]) > 0
for: 5m
labels:
  severity: critical
annotations:
  summary: "SYN queue overflow (instance {{ $labels.instance }})"

4.3 Stress Test Recommendation

Use wrk to simulate high load: wrk -t16 -c10000 -d60s http://service:8080 Key metrics to watch: SYN_RECV spikes, packet‑loss counters from netstat -s, and memory usage via free -m.

5. Pitfalls

5.1 Common Misconceptions

Blindly enabling tcp_tw_recycle breaks connections in NAT environments and has been removed since Linux 4.12.

Setting buffer sizes too large can cause OOM; adjust according to available memory (e.g., tcp_mem).

5.2 Parameter Dependencies

somaxconn

must be greater than or equal to the application’s backlog (e.g., Nginx listen 80 backlog=65535).

6. Conclusion

After applying the above tuning, the system achieved a 70 % reduction in TIME_WAIT connections, increased maximum concurrent connections to over 30 k, and doubled network throughput.