Linux System Tuning Golden Rules: Choosing the Right /proc/sys/net Core Parameters

Why blind sysctl tweaks often fail

Symptoms such as occasional connection timeouts, slow TCP handshakes, listen‑queue overflows, or exhaustion of short‑lived ports usually lead to the suggestion “increase the sysctl values”. Raising a parameter can relieve a symptom but often moves the bottleneck to another queue, so the root cause must be identified first.

/proc/sys/net and sysctl basics

/proc/sys/net

provides a runtime view of the Linux network stack; sysctl is the common read/write interface. The guide uses an Ubuntu Server managed by systemd as an example and stresses that kernel version, NIC driver, cloud load balancer, container runtime, NAT, application protocol and service connection model all affect the meaning of each parameter.

Golden rule: locate the queue or state before changing a parameter

A TCP request passes through the NIC, soft‑interrupt, kernel protocol stack, SYN queue, accept queue, file descriptor, worker, upstream connection and possibly NAT/conntrack. Each stage has its own capacity and drop policy. Only after confirming which stage is saturated should you decide whether to tune kernel parameters, application concurrency, load‑balancer settings, connection pools or add nodes.

Typical symptoms, evidence and relevant parameters

New‑connection occasional timeout : check listen queue, SYN statistics, application accept latency; relevant sysctls net.core.somaxconn, net.ipv4.tcp_max_syn_backlog; do not assume DDoS or that syncookies must be disabled.

Large backlog of established connections : examine ss state, application thread pool, upstream latency; relevant sysctls tcp_keepalive, buffer sizes, file‑descriptor limits; increasing all timeouts is not a guaranteed fix.

NAT‑induced intermittent packet loss : look at conntrack usage, packet drops, firewall rules; relevant sysctl net.netfilter.nf_conntrack_max; do not blindly enlarge the table.

High‑throughput packet loss : inspect NIC statistics, softnet, CPU interrupt distribution; relevant sysctls net.core.netdev_max_backlog, RPS/XPS; a single sysctl change rarely solves NIC bottlenecks.

Client‑side port exhaustion : check TIME‑WAIT, ESTABLISHED, port range; relevant sysctl net.ipv4.ip_local_port_range; expanding the range only postpones the problem if the application leaks ports.

Step‑by‑step evidence collection

1. Record host, kernel and network‑stack basics:

# Code 1: Record host, kernel and network stack basic info
uname -a
lsb_release -a
systemctl --version
sysctl -n kernel.osrelease
ip -br link
ip -br address

2. Read current values of the most common parameters:

# Code 2: Read current effective values of common network parameters
sysctl -n net.core.somaxconn
sysctl -n net.core.netdev_max_backlog
sysctl -n net.ipv4.tcp_max_syn_backlog
sysctl -n net.ipv4.tcp_syncookies
sysctl -n net.ipv4.ip_local_port_range
sysctl -n net.netfilter.nf_conntrack_max

3. Find which configuration files provide those values (search /etc/sysctl.conf, /etc/sysctl.d, /usr/lib/sysctl.d, /run/sysctl.d and use systemd-analyze cat-config sysctl.d).

Listen‑queue tuning

The backlog set by the application limits the listen queue; net.core.somaxconn is only an upper bound. If the application backlog is too small, increasing somaxconn has no effect. If the application accepts too slowly, a larger queue only lengthens client wait time.

# Code 4: View listen socket queue limits and current backlog
ss -ltnp
ss -ltn 'sport = :<port>'
ss -ltnH 'sport = :<port>' | awk '{print "recv_q="$2, "send_q="$3, "local="$4}'

Continuous sampling of Recv‑Q and Send‑Q together with CPU usage helps decide whether the problem is in accept() handling or elsewhere.

NIC, soft‑interrupt and receive‑side backpressure

When packet loss originates in the driver, ring buffer, soft‑interrupt or CPU‑interrupt distribution, merely raising net.core.netdev_max_backlog is ineffective. The article recommends checking NIC errors, drops and queue statistics with ip -s link and ethtool, and examining /proc/softnet_stat and /proc/softirqs for NET_RX/NET_TX activity.

# Code 10: Check NIC byte, error and drop counters
ip -s link show dev <nic_name>
ethtool <nic_name>
ethtool -S <nic_name> | head -n 120

Temporary port range and connection lifecycle

Outbound short connections, proxy back‑ends, service mesh and high‑frequency HTTP/1.1 clients consume local ports. net.ipv4.ip_local_port_range defines the automatic allocation range, but expanding it is not the first‑choice fix; connection‑pool leaks, TIME‑WAIT storms or NAT exhaustion must be addressed first.

# Code 13: View current temporary port range and TCP state summary
sysctl net.ipv4.ip_local_port_range
ss -s
ss -tan state established | wc -l
ss -tan state time-wait | wc -l

Conntrack table considerations

Stateful iptables/nftables rules, container NAT, L4 proxies or node‑level SNAT use the conntrack table. When the table is full, new connections fail and kernel logs show errors. The guide shows how to read net.netfilter.nf_conntrack_count and net.netfilter.nf_conntrack_max, and how to use conntrack -S for statistics before deciding to enlarge the table.

# Code 19: Check conntrack current count, limit and tool availability
sysctl net.netfilter.nf_conntrack_count
sysctl net.netfilter.nf_conntrack_max
command -v conntrack || true
dmesg --level=err,warn | rg -i 'conntrack|nf_conntrack' || true

Buffers, queues and congestion control

Socket buffer limits ( net.core.rmem_max, net.core.wmem_max, net.ipv4.tcp_rmem, net.ipv4.tcp_wmem) help high‑throughput long‑lived flows but increase per‑connection memory usage. The default qdisc and congestion‑control algorithm also affect latency; the article shows how to enable fq and bbr when supported.

# Code 23: Enable fq and bbr when kernel support is verified
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Verification, observation and rollback

Any change must be falsifiable: compare the same traffic window before and after, measuring request volume, connection errors, retransmissions, queue overflows, application latency, CPU, memory, conntrack usage and downstream error rate. Example Prometheus queries are provided for TCP TIME‑WAIT count and receive‑drop rate.

# Code 25: Example metric for TCP TIME‑WAIT count (node exporter)
node_sockstat_TCP_tw

Rollback steps include backing up the original sysctl values and configuration files, restoring them, re‑loading with sysctl -p (or sysctl --system only after full validation), and re‑checking the same set of metrics.

Final takeaway

Linux network parameters have no universal “golden numbers”. The proper workflow is to gather evidence, map the symptom to the exact queue or state, apply the smallest possible change, and verify the effect with identical load. Unverifiable or irreversible changes must never become part of a production baseline.