Master Linux Network Tuning for Million‑Connection High Concurrency

Linux High‑Concurrency Network Parameter Tuning Guide

Introduction

In high‑concurrency network services, the default Linux kernel network parameters often become a performance bottleneck, causing connection timeouts or service crashes. This article presents a real‑world case study, explains key parameters, diagnoses issues, and provides step‑by‑step tuning practices to support millions of concurrent connections.

1. Problem Background: When Concurrency Hits Performance Limits

1.1 Case Environment

Server configuration:

vCPU: 8 cores | Memory: 16GB | Bandwidth: 4Gbps | PPS: 800,000

Observed anomalies: TIME_WAIT connections piled up (2464 instances) CLOSE_WAIT connections (4 instances)

Occasional new‑connection establishment timeouts

1.2 Initial Parameter Analysis

Using sysctl we see the original configuration:

net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 131072
net.ipv4.ip_local_port_range = 1024 61999

Key defects: small half‑open queue, narrow port range, and strict buffer limits.

2. Deep Diagnosis: Connection States and Kernel Parameters

2.1 Connection State Monitoring Techniques

Real‑time TCP State Statistics

watch -n 1 'netstat -ant | awk "/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

Sample output:

ESTABLISHED 790
TIME_WAIT 2464
SYN_RECV 32  # half‑open connections to watch

Half‑Open Connection Check

# Show SYN_RECV details
ss -ntp state syn-recv
# Monitor listen queue drops
netstat -s | grep -i 'listen drops'

2.2 Key Parameter Interpretation

tcp_max_syn_backlog

: half‑open queue length, default 8192 – may fill under burst traffic. somaxconn: full‑connection queue length, should match application backlog. tcp_tw_reuse: enables rapid reuse of TIME_WAIT ports; disabled by default, leading to port exhaustion. tcp_rmem / tcp_wmem: read/write buffer sizes; maximum only 6 MB, limiting throughput.

3. Tuning Solutions: From Parameters to Practice

3.1 Connection Management Optimization

Resolve TIME_WAIT Accumulation

echo "net.ipv4.tcp_tw_reuse = 1" >> /etc/sysctl.conf
echo "net.ipv4.tcp_max_tw_buckets = 262144" >> /etc/sysctl.conf
echo "net.ipv4.ip_local_port_range = 1024 65000" >> /etc/sysctl.conf

Shorten Connection Recycle Time

echo "net.ipv4.tcp_fin_timeout = 30" >> /etc/sysctl.conf

3.2 Queue and Buffer Optimization

Expand Connection Queues

echo "net.ipv4.tcp_max_syn_backlog = 65535" >> /etc/sysctl.conf
echo "net.core.somaxconn = 65535" >> /etc/sysctl.conf
echo "net.core.netdev_max_backlog = 10000" >> /etc/sysctl.conf

Adjust Memory Buffers

cat >> /etc/sysctl.conf <<EOF
net.ipv4.tcp_mem = 8388608 12582912 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
EOF

3.3 Keepalive and Timeout Optimization

echo "net.ipv4.tcp_keepalive_time = 600" >> /etc/sysctl.conf
echo "net.ipv4.tcp_keepalive_intvl = 30" >> /etc/sysctl.conf

4. Validation and Monitoring

4.1 Real‑time Monitoring Script

#!/bin/bash
while true; do
  clear
  date
  echo "---- TCP State ----"
  netstat -ant | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
  echo "---- Half‑Open Queue ----"
  ss -ltn | awk 'NR>1 {print "Listen queue: Recv-Q="$2", Send-Q="$3}'
  echo "---- Port Usage ----"
  echo "Used ports: $(netstat -ant | grep -v LISTEN | awk '{print $4}' | cut -d: -f2 | sort -u | wc -l)/$((65000-1024))"
  sleep 5
done

Prometheus Alert Example for SYN Queue Overflow

alert: TCP_SYN_Dropped
expr: increase(node_netstat_Tcp_Ext_SyncookiesFailed{job="node"}[1m]) > 0
for: 5m
labels:
  severity: critical
annotations:
  summary: "SYN queue overflow (instance {{ $labels.instance }})"

4.2 Load‑Testing Recommendations

Use wrk to simulate high concurrency: wrk -t16 -c10000 -d60s http://service:8080 Key metrics to monitor during tests: SYN_RECV count fluctuations

Packet loss counters from netstat -s Memory usage via

free -m

5. Pitfall Guide

5.1 Common Misconceptions

Blindly enabling tcp_tw_recycle breaks NAT environments and has been removed since Linux 4.12.

Oversized buffers can cause OOM; adjust tcp_mem based on available RAM.

5.2 Parameter Dependencies

somaxconn

must be greater than or equal to the application’s backlog (e.g., Nginx listen 80 backlog=65535).

6. Conclusion

By applying the above tuning steps, we achieved:

70% reduction in TIME_WAIT connections.

Maximum concurrent connections exceeding 30,000.

Network throughput doubled.