Why Does TCP Send‑Q Exceed SO_SNDBUF? Inside Linux Kernel Buffer Mechanics

Test scenario

A client creates a TCP socket, sets SO_SNDBUF to 4096 bytes, and sends a 1024‑byte segment every second to a server that never calls recv(). The expected behaviour consists of three phases:

Phase 1 – the server’s receive buffer is not full, so ACKs are still sent.

Phase 2 – the server’s receive buffer fills, it advertises a zero‑window, and the client’s send buffer starts to accumulate data.

Phase 3 – the client’s send buffer becomes full and send() blocks.

Monitoring the connection with ss -nt shows the Send‑Q growing from 0 to 14480, far exceeding the configured SO_SNDBUF of 4096.

Why SO_SNDBUF appears doubled

When a user sets SO_SNDBUF, the kernel stores val * 2 in sk->sk_sndbuf. The multiplication reserves space for internal overhead such as sk_buff structures and protocol headers. Thus a request of 4096 bytes is recorded as 8192 bytes.

Memory accounting with sk_wmem_queued

The kernel tracks the actual memory used by the send queue in sk->wmem_queued. This value equals the memory occupied by pending data plus the overhead of each sk_buff, so it is always greater than the visible Send‑Q.

bool sk_stream_memory_free(const struct sock *sk) {
    if (sk->sk_wmem_queued >= sk->sk_sndbuf)
        return false;
    ...
}

tcp_sendmsg workflow

tcp_sendmsg

decides whether to allocate a new sk_buff or to append data to the last one in the write queue.

Case 1 – allocate a new sk_buff

During Phase 1 each 1 KB segment creates a fresh sk_buff via sk_stream_alloc_skb. The kernel first checks sk_stream_memory_free, which succeeds because the doubled sk_sndbuf (8192) is still larger than the current usage.

Case 2 – append to the last sk_buff

In Phase 2 the client already has accumulated sk_buff s, so new data is appended to the tail sk_buff. The amount that can be appended is calculated as:

int max = size_goal;
copy = max - skb->len;

How size_goal is computed

The kernel calls tcp_send_mss → tcp_xmit_size_goal. If Generic Segmentation Offload (GSO) is enabled, size_goal = tp->gso_segs * mss_now If GSO is disabled, size_goal = mss_now In the observed environment the effective MSS is 1448 bytes and GSO yields size_goal = 14480 bytes (10 × MSS). Consequently, when Phase 2 begins, tcp_sendmsg computes copy = 14480 - 1024 = 13456 bytes, far larger than the original 4096‑byte buffer.

Write‑queue expansion via sk_wmem_schedule

Before copying data, tcp_sendmsg invokes sk_wmem_schedule. Inside __sk_mem_schedule the kernel adds SK_MEM_QUANTUM chunks to sk_forward_alloc and updates accounting, allowing sk_wmem_queued to exceed sk_sndbuf when system memory permits.

if (!sk_wmem_schedule(sk, copy))
    goto wait_for_memory;

This mechanism makes the user‑specified SO_SNDBUF effectively non‑binding in the presence of GSO.

Possible mitigations

Disable GSO on the network interface (e.g., ethtool -K eth0 gso off).

Patch the kernel to move the sk_stream_memory_free check to the beginning of the while (msg_data_left(msg)) loop in tcp_sendmsg, preventing the write queue from growing beyond the advertised limit.