When Does Socket Buffer Size Actually Impact TCP Performance? A Practical Linux Experiment

Impact of Socket Buffer Parameters

Linux provides four core sysctl files that control the default and maximum sizes of socket receive and send buffers:

/proc/sys/net/core/rmem_default /proc/sys/net/core/rmem_max /proc/sys/net/core/wmem_default /proc/sys/net/core/wmem_max

These affect both TCP and UDP. For TCP the same four files are used, but the effective defaults are taken from /proc/sys/net/ipv4/tcp_rmem and /proc/sys/net/ipv4/tcp_wmem, each containing three values (min default max).

Experiment Setup

Two virtual machines are used: a CentOS 8 host running a simple web server that serves a 1 GiB file, and a Fedora 31 client that downloads the file with wget. The baseline network is unrestricted, giving a raw transfer rate of about 337 MiB/s between the VMs.

To isolate the effect of the socket buffer, traffic on port 80 is limited with an HTB qdisc script ( htb.sh) that sets a 1000 Mbit/s rate for port 80 and a very high rate for other traffic.

Baseline Results (No Added Delay)

[root@zorro]# wget --no-proxy http://192.168.247.129/bigfile
Length: 1073741824 (1.0G)   337MiB/s   3.0s

With the 1000 Mbit/s limit the download speed matches the limit, confirming the test environment works.

Adding Network Delay

Using tc netem delay 200ms on the HTB class for port 80 introduces a 200 ms one‑way delay, which appears as ~400 ms RTT in httping output. The same file download now yields only about 13 MiB/s:

[root@zorro]# wget --no-proxy http://192.168.247.129/bigfile
Length: 1073741824 (1.0G)   13.4MiB/s   87s

The bandwidth is no longer fully utilized because the sender must wait for acknowledgments before sending more data.

Bandwidth‑Delay Product Analysis

With a 1000 Mbit/s link (≈125 MiB/s) and 400 ms RTT, the bandwidth‑delay product is 125 MiB/s × 0.4 s ≈ 50 MiB. To keep the pipe full, the TCP send buffer should be at least this size.

Adjusting tcp_wmem (Sender) and tcp_rmem (Receiver)

On the server (CentOS 8) the send buffer is increased:

echo 52428800 52428800 52428800 > /proc/sys/net/ipv4/tcp_wmem

Resulting download speed rises to ~15 MiB/s, still below the theoretical maximum because the receiver’s advertised window is still limited.

On the client (Fedora 31) the receive buffer is increased similarly:

echo 52428800 52428800 52428800 > /proc/sys/net/ipv4/tcp_rmem

Now the transfer reaches ~92 MiB/s, close to the expected value based on the 50 MiB pipe size.

Further increasing the receiver buffers to 100 MiB (double the BDP) gives a stable ~89 MiB/s transfer, confirming that both sides must have sufficiently large buffers.

Additional Tuning Parameters

The file /proc/sys/net/ipv4/tcp_adv_win_scale controls how much of the buffer is reserved for TCP overhead. Its default value (1) reserves half of the buffer for overhead; adjusting it can fine‑tune the effective window size.

During the tests the ss -io command shows the evolving rcv_space and rcv_ssthresh values, illustrating how the kernel advertises the receive window based on remaining buffer space.

Conclusion

On high‑latency links the socket buffer size directly limits TCP throughput. The practical rule is to set the sender’s tcp_wmem and the receiver’s tcp_rmem to at least the bandwidth‑delay product (BDP). In the example, a 1000 Mbit/s link with 400 ms RTT required roughly 50 MiB of send buffer and 100 MiB of receive buffer to achieve near‑line‑rate performance. Real‑world networks may have additional bottlenecks such as intermediate device buffers or packet loss, so congestion‑control algorithm choice (e.g., BBR) can also affect the final throughput.