Master Linux Network Performance: Practical TCP/IP Stack Tuning Guide

Linux Network Performance Tuning: TCP/IP Stack Parameter Optimization Practice

🚀 Introduction: Why Network Performance Optimization Is Essential

In today’s high‑concurrency, high‑traffic Internet environment, network performance often becomes the system bottleneck. As an experienced operations engineer, I have repeatedly encountered performance issues caused by improper TCP/IP parameter settings in production. This article shares a complete Linux network performance tuning solution to help eliminate network bottlenecks.

📊 Common Manifestations of Network Performance Issues

High‑concurrency connection scenarios : During e‑commerce flash sales, server connections surge, leading to many TIME_WAIT states.

Large file transfer scenarios : Data backup suffers from insufficient network throughput and low transfer efficiency.

Micro‑service call scenarios : Frequent inter‑service calls cause latency jitter and unstable response times.

These problems usually stem from the Linux kernel’s default TCP/IP parameters not meeting high‑performance demands.

🔧 Core TCP Stack Parameter Optimization

1. TCP Connection Management Optimization

# /etc/sysctl.conf configuration file
# TCP connection queue length optimization
net.core.somaxconn = 65535            # increase listen queue length
net.core.netdev_max_backlog = 30000   # NIC receive queue length
net.ipv4.tcp_max_syn_backlog = 65535  # SYN queue length

# TIME_WAIT state optimization
net.ipv4.tcp_tw_reuse = 1            # allow reuse of TIME_WAIT sockets
net.ipv4.tcp_fin_timeout = 30        # reduce FIN_WAIT_2 duration
net.ipv4.tcp_max_tw_buckets = 10000  # limit TIME_WAIT count

# Connection keepalive
net.ipv4.tcp_keepalive_time = 600    # start keepalive probes after 600s
net.ipv4.tcp_keepalive_probes = 3     # number of keepalive probes
net.ipv4.tcp_keepalive_intvl = 15     # interval between probes

2. TCP Buffer Optimization

# TCP receive/send buffer optimization
net.core.rmem_default = 262144
net.core.rmem_max = 16777216
net.core.wmem_default = 262144
net.core.wmem_max = 16777216

# TCP socket buffer auto‑tuning
net.ipv4.tcp_rmem = 4096 87380 16777216   # min default max
net.ipv4.tcp_wmem = 4096 65536 16777216   # min default max
net.ipv4.tcp_mem = 94500000 915000000 927000000

# Enable TCP window scaling
net.ipv4.tcp_window_scaling = 1

3. TCP Congestion Control Optimization

# Choose congestion control algorithm
net.ipv4.tcp_congestion_control = bbr   # recommended BBR algorithm
# Other options: cubic, reno, bic

# Fast retransmit and recovery
net.ipv4.tcp_frto = 2   # F‑RTO detects spurious timeouts
net.ipv4.tcp_dsack = 1   # enable DSACK support
net.ipv4.tcp_fack = 1    # enable FACK congestion avoidance

# TCP slow start after idle
net.ipv4.tcp_slow_start_after_idle = 0   # disable slow start after idle

🌐 IP Stack Parameter Optimization

1. IP Layer Processing Optimization

# IP forwarding and routing optimization
net.ipv4.ip_forward = 0               # disable forwarding on non‑router devices
net.ipv4.conf.default.rp_filter = 1    # enable reverse path filtering
net.ipv4.conf.all.rp_filter = 1

# IP fragmentation handling
net.ipv4.ipfrag_high_thresh = 262144    # high threshold
net.ipv4.ipfrag_low_thresh = 196608     # low threshold
net.ipv4.ipfrag_time = 30               # reassembly timeout

# ICMP optimization
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1

2. Port Range Optimization

# Expand local port range
net.ipv4.ip_local_port_range = 1024 65535

# UDP port optimization
net.ipv4.udp_mem = 94500000 915000000 927000000
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192

⚡ Network Queue and Interrupt Optimization

1. Network Device Queue Optimization

# Increase network device processing queues
echo 'echo 4096 > /proc/sys/net/core/netdev_budget' >> /etc/rc.local
echo 'echo 2 > /proc/sys/net/core/netdev_budget_usecs' >> /etc/rc.local

# RPS/RFS optimization (multi‑core CPU load balancing)
echo 'f' > /sys/class/net/eth0/queues/rx-0/rps_cpus  # adjust according to CPU cores

2. Interrupt Optimization Script

#!/bin/bash
# network_irq_balance.sh - network interrupt balancing script

# Get NIC interrupt numbers
IRQ_LIST=$(grep eth0 /proc/interrupts | awk -F: '{print $1}' | xargs)

# Bind interrupts to different CPU cores
CPU_COUNT=$(nproc)
i=0
for irq in $IRQ_LIST; do
    cpu_mask=$((1 << (i % CPU_COUNT)))
    printf "%x" $cpu_mask > /proc/irq/$irq/smp_affinity
    echo "IRQ $irq -> CPU $((i % CPU_COUNT))"
    ((i++))
done

🎯 High‑Concurrency Scenario Specialized Optimization

1. Large Connection Count Optimization

# File descriptor limits
echo '* soft nofile 1048576' >> /etc/security/limits.conf
echo '* hard nofile 1048576' >> /etc/security/limits.conf

# Process limits
echo '* soft nproc 1048576' >> /etc/security/limits.conf
echo '* hard nproc 1048576' >> /etc/security/limits.conf

# systemd service limits
echo 'DefaultLimitNOFILE=1048576' >> /etc/systemd/system.conf
echo 'DefaultLimitNPROC=1048576' >> /etc/systemd/system.conf

2. Memory Management Optimization

# Virtual memory management
vm.swappiness = 10               # reduce swap usage
vm.dirty_ratio = 15             # dirty page write‑back ratio
vm.dirty_background_ratio = 5  # background write‑back ratio
vm.overcommit_memory = 1       # allow memory over‑commit

📈 Performance Monitoring and Validation

1. Key Metric Monitoring Script

#!/bin/bash
# network_monitor.sh - network performance monitoring

echo "=== Network Connection Status Summary ==="
ss -s

echo -e "
=== TCP Connection State Distribution ==="
ss -tan | awk 'NR>1{state[$1]++} END{for(i in state) print i, state[i]}'

echo -e "
=== Network Throughput ==="
sar -n DEV 1 1 | grep -E "eth0|Average"

echo -e "
=== Memory Usage ==="
free -h

echo -e "
=== System Load ==="
uptime

2. Stress Test Verification Commands

# HTTP load test with wrk
wrk -t12 -c400 -d30s --latency http://your-server-ip/

# Bandwidth test with iperf3
iperf3 -s               # server side
iperf3 -c server-ip -t 60 -P 10   # client side

# TCP connection load test
ab -n 100000 -c 1000 http://your-server-ip/

🔥 Real‑World Case: E‑Commerce System Optimization

Key Optimization Points

BBR congestion control : Enables a 40% increase in network throughput.

TCP buffer tuning : Significantly reduces network latency jitter.

Connection reuse optimization : Decreases TIME_WAIT states by 90%.

Interrupt balancing : Improves multi‑core CPU utilization.

💡 Best‑Practice Recommendations

1. Scenario‑Specific Tuning Strategies

High‑concurrency web servers

# Focus on connection count and fast release
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

Large file transfer servers

# Emphasize buffer and window size
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_window_scaling = 1

Database servers

# Optimize keepalive and stability
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_retries2 = 5

2. Production Deployment Process

Test environment verification : Apply configurations in a staging environment first.

Canary release : Deploy to a subset of servers initially.

Monitoring observation : Closely monitor key performance indicators.

Full rollout : Promote to all servers after confirming stability.

3. Configuration Persistence

# Apply all sysctl settings
sysctl -p

# Verify applied settings
sysctl net.ipv4.tcp_congestion_control
sysctl net.core.somaxconn

# Ensure settings survive reboot
echo 'sysctl -p' >> /etc/rc.local
chmod +x /etc/rc.local

⚠️ Cautions and Common Pitfalls

1. Parameter Tuning Misconceptions

Blindly increasing buffers : May cause memory exhaustion.

Over‑optimizing TIME_WAIT : Can lead to port exhaustion.

Ignoring business characteristics : Different workloads require different parameter strategies.

2. Rollback Plan

# Backup current configuration
cp /etc/sysctl.conf /etc/sysctl.conf.backup.$(date +%Y%m%d)

# Quick rollback script
cat > /root/network_rollback.sh <<'EOF'
#!/bin/bash
cp /etc/sysctl.conf.backup.* /etc/sysctl.conf
sysctl -p
echo "Network config rollback completed!"
EOF
chmod +x /root/network_rollback.sh

🎓 Summary

Through systematic TCP/IP stack parameter tuning, we can significantly improve Linux server network performance. The key points are:

Understand business characteristics : Choose appropriate optimization strategies based on actual scenarios.

Iterative tuning : Modify parameters gradually to facilitate issue isolation.

Continuous monitoring : Build a comprehensive monitoring system to detect performance problems promptly.

Test verification : Conduct thorough performance testing after each tuning step.