Unlocking Millions of TCP Connections: Deep Dive into Linux Socket Internals and Tuning

The article begins by expanding the default local port range on a Linux system using the command echo "5000 65000" > /proc/sys/net/ipv4/ip_local_port_range, allowing ports up to 65000 for outbound connections.

It then demonstrates how to increase the maximum number of file descriptors the kernel can handle, for example:

echo 200000 > /proc/sys/fs/file-max
# vi /etc/sysctl.conf
fs.nr_open=210000
# sysctl -p
# vi /etc/security/limits.conf
*  soft  nofile  200000
*  hard  nofile  200000

Note: the hard limit in limits.conf cannot exceed nr_open , so nr_open must be increased first, preferably in sysctl.conf , to avoid startup failures after a reboot.

Next, the article examines the core kernel data structure for sockets, struct sock_common, which contains two unions: one for the IP address pair ( skc_addrpair) and one for the port pair ( skc_portpair). The relevant excerpt from include/net/sock.h is shown:

struct sock_common {
    union {
        __addrpair skc_addrpair; // TCP connection IP pair
        struct {
            __be32 skc_daddr;
            __be32 skc_rcv_saddr;
        };
    };
    union {
        __portpair skc_portpair; // TCP connection port pair
        struct {
            __be16 skc_dport;
            __u16  skc_num;
        };
    };
    ...
};

The article then follows the packet processing path: after a network card receives a packet, it goes through DMA, hard interrupt, soft interrupt, and finally lands in the socket's receive queue.

For TCP packets, the entry point in the kernel is tcp_v4_rcv (file

net/ipv4/tcp_ipv4.c</code):</p>
<pre><code>int tcp_v4_rcv(struct sk_buff *skb) {
    ...
    th = tcp_hdr(skb); // get TCP header
    iph = ip_hdr(skb); // get IP header
    sk = __inet_lookup_skb(&tcp_hashinfo, skb, th->source, th->dest);
    ...
}

The lookup routine eventually calls __inet_lookup_established , which builds a 32‑bit port key and hashes it to locate the appropriate struct sock in the hash table:

static inline struct sock *__inet_lookup_established(struct net *net,
        struct inet_hashinfo *hashinfo,
        const __be32 saddr, const __be16 sport,
        const __be32 daddr, const u16 hnum,
        const int dif)
{
    const __portpair ports = INET_COMBINED_PORTS(sport, hnum);
    unsigned int hash = inet_ehashfn(net, daddr, hnum, saddr, sport);
    unsigned int slot = hash & hashinfo->ehash_mask;
    struct inet_ehash_bucket *head = &hashinfo->ehash[slot];
    sk_nulls_for_each_rcu(sk, node, &head->chain) {
        if (sk->sk_hash != hash)
            continue;
        if (likely(INET_MATCH(sk, net, acookie, saddr, daddr, ports, dif))) {
            if (unlikely(!atomic_inc_not_zero(&sk->sk_refcnt)))
                goto begintw;
            if (unlikely(!INET_MATCH(sk, net, acookie, saddr, daddr, ports, dif))) {
                sock_put(sk);
                goto begin;
            }
            goto out;
        }
    }
}

The macro INET_MATCH compares the packet’s source/destination addresses and ports with the socket’s stored values, ensuring a correct four‑tuple match:

#define INET_MATCH(__sk, __net, __cookie, __saddr, __daddr, __ports, __dif) \
    ((inet_sk(__sk)->inet_portpair == (__ports)) && \
     (inet_sk(__sk)->inet_daddr == (__saddr)) && \
     (inet_sk(__sk)->inet_rcv_saddr == (__daddr)) && \
     (!(__sk)->sk_bound_dev_if || (__sk)->sk_bound_dev_if == (__dif)) && \
     net_eq(sock_net(__sk), (__net)))

System diagnostics shown include the OS version, the number of established sockets ( ss -ant | grep ESTAB | wc -l returning over one million), and memory statistics from /proc/meminfo , illustrating the resource pressure of massive connections.

Summary

Each client connection consumes one local port; when the number of connections exceeds 30‑50 k, administrators often worry about limits. The article clarifies that a TCP connection is identified by a four‑tuple (client IP, client port, server IP, server port) stored in kernel socket objects. To increase the client‑side concurrency, two approaches are recommended: assign multiple IP addresses to the client host, or connect to many distinct servers. Mixing both is discouraged because binding a specific IP changes the kernel’s port‑selection strategy. Practical experiments confirm that, after applying the described sysctl and limits adjustments, a Linux client can sustain well over one million simultaneous TCP connections, effectively dispelling the myth that the 65535 port limit is an absolute barrier.