What Really Happens Inside listen()? Uncovering Linux Kernel’s Connection Queues

1. Creating a socket

The server starts by calling socket(), which returns a file descriptor in user space but actually creates a complex kernel object composed of several nested structures.

2. Kernel execution of listen

In net/socket.c the listen syscall performs the following steps:

sock = sockfd_lookup_light(fd, &err, &fput_needed);
if (sock) {
    somaxconn = sock_net(sock->sk)->core.sysctl_somaxconn;
    if ((unsigned int)backlog > somaxconn)
        backlog = somaxconn;
    err = sock->ops->listen(sock, backlog);
    ...
}

It looks up the kernel socket object from the user‑space fd, clamps the supplied backlog to the kernel parameter net.core.somaxconn, and then delegates to the protocol‑stack specific listen implementation.

2.1 Protocol‑stack listen

The generic socket layer calls inet_listen (in net/ipv4/af_inet.c), which sets the maximum length of the full‑connection queue and invokes inet_csk_listen_start:

if (old_state != TCP_LISTEN) {
    err = inet_csk_listen_start(sk, backlog);
}
sk->sk_max_ack_backlog = backlog;

2.2 inet_csk_listen_start

This function casts the generic sock to an inet_connection_sock and allocates the request‑socket queue:

int rc = reqsk_queue_alloc(&icsk->icsk_accept_queue, nr_table_entries);

The inet_connection_sock contains an inet_sock and a request_sock_queue, which holds both the full‑connection (accept) queue and the half‑connection (listen) queue.

2.3 Definition of the receive queues

The request_sock_queue structure looks like:

struct request_sock_queue {
    struct request_sock *rskq_accept_head;
    struct request_sock *rskq_accept_tail;
    struct listen_sock *listen_opt; // half‑connection queue
    ...
};

The full‑connection queue is a simple FIFO list managed by rskq_accept_head and rskq_accept_tail. The half‑connection queue is represented by listen_opt, a listen_sock object.

2.4 listen_sock layout

struct listen_sock {
    u8  max_qlen_log;
    u32 nr_table_entries;
    struct request_sock *syn_table[0]; // hash table for half‑connections
    ...
};

The hash table syn_table enables fast lookup of the request_sock created during the TCP three‑way handshake.

2.5 Allocation and initialization of the receive queue

The function reqsk_queue_alloc (in net/core/request_sock.c) performs the heavy lifting:

nr_table_entries = min_t(u32, nr_table_entries, sysctl_max_syn_backlog);
nr_table_entries = max_t(u32, nr_table_entries, 8);
nr_table_entries = roundup_pow_of_two(nr_table_entries + 1);
/* allocate listen_sock */
if (lopt_size > PAGE_SIZE)
    lopt = vzalloc(lopt_size);
else
    lopt = kzalloc(lopt_size, GFP_KERNEL);
queue->rskq_accept_head = NULL;
queue->listen_opt = lopt;

It clamps the requested size to sysctl_max_syn_backlog, enforces a minimum of 8, rounds up to the next power of two, allocates memory for listen_sock, and initializes both the full‑connection and half‑connection structures.

If you encounter a full‑connection queue overflow in production, you must increase both the backlog argument and the net.core.somaxconn kernel parameter.

2.6 How the half‑connection length is computed

The effective half‑connection queue length is derived from the smallest of three values, then adjusted:

Take min(backlog, somaxconn).

Clamp again with sysctl_max_syn_backlog (often tcp_max_syn_backlog).

Ensure the result is at least 8.

Round up to the next power of two; the minimum final size is 16.

In formula form:

half_len = max(16, roundup_pow_of_two(min(backlog, somaxconn, tcp_max_syn_backlog) + 1));

2.7 Example calculations

Assume net.core.somaxconn = 128 and net.ipv4.tcp_max_syn_backlog = 8192:

With backlog = 5 the half‑connection length becomes 16.

With backlog = 512 the half‑connection length becomes 256.

Online articles that suggest changing only one kernel parameter to enlarge the half‑connection queue are misleading; you must consider backlog , somaxconn , and tcp_max_syn_backlog together.

3. Final summary

The listen system call primarily allocates and initializes two queues:

Full‑connection queue : a FIFO list whose maximum length is min(backlog, net.core.somaxconn).

Half‑connection queue : a hash table whose size follows the formula above, with a minimum of 16 entries.

These structures are essential for handling the TCP three‑way handshake; without them the server cannot accept incoming connections.

Understanding the exact calculations helps you tune the kernel parameters correctly and avoid common pitfalls that lead to connection‑queue overflows.