Boost Linux TCP Connections: Tuning Port Ranges, File Descriptors, and Kernel Socket Structures

Overview

The article demonstrates practical steps to raise the maximum number of TCP connections a single Linux host can handle, explains the kernel data structures involved, and provides concrete commands for tuning the system.

Adjusting Port Range

Increase the local port range so the client can allocate more source ports:

echo "5000 65000" > /proc/sys/net/ipv4/ip_local_port_range

Example connections after the change range from 5000 to 65000, allowing up to 60,000 distinct ports per client IP.

Increasing File Descriptor Limits

Raise the global file descriptor limit and per‑process limits:

// Set system‑wide maximum to 200,000
 echo 200000 > /proc/sys/fs/file-max

// In /etc/sysctl.conf
fs.nr_open=210000

// Apply the change
sysctl -p

// In /etc/security/limits.conf
* soft nofile 200000
* hard nofile 200000

Note: The hard limit in limits.conf cannot exceed nr_open , so modify nr_open first and preferably keep the change in sysctl.conf to avoid boot‑time inconsistencies.

Kernel Socket Structure sock_common

The core socket data structure contains two unions that store the IP address pair and the port pair of a TCP connection:

struct sock_common {
    union {
        __addrpair skc_addrpair; // IP pair
        struct {
            __be32 skc_daddr;
            __be32 skc_rcv_saddr;
        };
    };
    union {
        __portpair skc_portpair; // Port pair
        struct {
            __be16 skc_dport;
            __u16  skc_num;
        };
    };
    ...
};

skc_addrpair

records the source and destination IPs, while skc_portpair records the source and destination ports.

Packet Reception Path

When a packet arrives at the NIC it passes through DMA, hard interrupt, soft interrupt, and finally lands in the socket’s receive queue. The entry point for TCP packet processing is tcp_v4_rcv:

int tcp_v4_rcv(struct sk_buff *skb) {
    ...
    th  = tcp_hdr(skb);   // TCP header
    iph = ip_hdr(skb);    // IP header
    sk  = __inet_lookup_skb(&tcp_hashinfo, skb, th->source, th->dest);
    ...
}

The lookup uses __inet_lookup_skb, which eventually calls __inet_lookup_established to find an existing socket based on the five‑tuple.

static inline struct sock *__inet_lookup(struct net *net,
                                         struct inet_hashinfo *hashinfo,
                                         const __be32 saddr, const __be16 sport,
                                         const __be32 daddr, const __be16 dport,
                                         const int dif) {
    u16 hnum = ntohs(dport);
    struct sock *sk = __inet_lookup_established(net, hashinfo,
                                                saddr, sport, daddr, hnum, dif);
    return sk ? sk : __inet_lookup_listener(net, hashinfo, saddr, sport,
                                            daddr, hnum, dif);
}

The macro INET_MATCH compares the packet’s source/destination addresses and ports with the socket’s stored values:

#define INET_MATCH(__sk, __net, __cookie, __saddr, __daddr, __ports, __dif) \
    ((inet_sk(__sk)->inet_portpair == (__ports)) && \
     (inet_sk(__sk)->inet_daddr == (__saddr)) && \
     (inet_sk(__sk)->inet_rcv_saddr == (__daddr)) && \
     (!(__sk)->sk_bound_dev_if || (__sk)->sk_bound_dev_if == (__dif)) && \
     net_eq(sock_net(__sk), (__net)))

This comparison forms the basis of the “five‑tuple” (source IP, source port, destination IP, destination port, protocol) used to identify a TCP connection.

Real‑World Verification

On a Red Hat Enterprise Linux 6.2 system the following commands show that more than one million ESTABLISHED connections can be achieved after the tuning:

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.2 (Santiago)

# ss -ant | grep ESTAB | wc -l
1000013

# cat /proc/meminfo
MemTotal:   3925408 kB
MemFree:     97748 kB
Buffers:      35412 kB
Cached:     119600 kB
Slab:      3241528 kB
...

Key Takeaways

Each TCP connection consumes one client port; the client’s port is only one element of the four‑tuple that uniquely identifies a connection.

Increasing the client’s concurrent connection capacity can be done by (a) assigning multiple IP addresses to the client or (b) connecting to many different servers.

Mixing the two methods is discouraged because binding to a specific IP changes the kernel’s port‑selection strategy.

With the described kernel tweaks, a single Linux client can handle well over a million concurrent TCP connections, far beyond the traditional 65,535 limit.

By understanding and adjusting the kernel parameters and socket lookup logic, developers and system administrators can reliably scale TCP workloads on Linux.