How to Enable One Million Simultaneous Connections on Linux: Limits, Memory & Tuning

How to Enable One Million Simultaneous Connections on Linux

By default most Linux kernels limit the number of open files, preventing a million (C1000K) connections. This guide explains how to lift global and per‑process limits, estimate memory requirements, test with sample server/client programs, and assess network bandwidth needs.

1. Can the OS support a million connections?

Linux imposes both global and per‑process file descriptor limits that must be increased.

Global limit

Check the current global maximum with: cat /proc/sys/fs/file-nr The third number (e.g., 101747) shows the current global maximum open files, roughly 100 k. To raise it, edit /etc/sysctl.conf and add:

fs.file-max = 1020000
net.ipv4.ip_conntrack_max = 1020000
net.ipv4.netfilter.ip_conntrack_max = 1020000

Process limit

Check the per‑process limit with: ulimit -n Typical output is 1024, meaning each process can open only 1024 files. To increase temporarily: ulimit -n 1020000 If you are not root, this will fail with “Operation not permitted”. For a permanent change, edit /etc/security/limits.conf:

# /etc/security/limits.conf
work         hard    nofile      1020000
work         soft    nofile      1020000

Replace work with * or root as needed, then re‑login.

Note that the kernel constant NR_OPEN in /usr/include/linux/fs.h caps the maximum; on RHEL 5 it is 1 048 576, so recompiling the kernel may be required for C1000K.

2. How much memory does the OS need for a million connections?

After lifting limits, memory usage becomes the next concern. A socket descriptor occupies a few bytes; estimating about 2 KB per connection, a million connections require roughly 2 GB of kernel memory.

Test program (server)

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <arpa/inet.h>
#include <netinet/tcp.h>
#include <sys/select.h>

#define MAX_PORTS 10

int main(int argc, char **argv){
    struct sockaddr_in addr;
    const char *ip = "0.0.0.0";
    int opt = 1;
    int connections = 0;
    int base_port = 7000;
    if(argc > 2){
        base_port = atoi(argv[1]);
    }

    int server_socks[MAX_PORTS];

    for(int i=0; i<MAX_PORTS; i++){
        int port = base_port + i;
        bzero(&addr, sizeof(addr));
        addr.sin_family = AF_INET;
        addr.sin_port = htons((short)port);
        inet_pton(AF_INET, ip, &addr.sin_addr);

        int serv_sock;
        if((serv_sock = socket(AF_INET, SOCK_STREAM, 0)) == -1){
            goto sock_err;
        }
        if(setsockopt(serv_sock, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)) == -1){
            goto sock_err;
        }
        if(bind(serv_sock, (struct sockaddr *)&addr, sizeof(addr)) == -1){
            goto sock_err;
        }
        if(listen(serv_sock, 1024) == -1){
            goto sock_err;
        }

        server_socks[i] = serv_sock;
        printf("server listen on port: %d
", port);
    }

    while(1){
        fd_set readset;
        FD_ZERO(&readset);
        int maxfd = 0;
        for(int i=0; i<MAX_PORTS; i++){
            FD_SET(server_socks[i], &readset);
            if(server_socks[i] > maxfd){
                maxfd = server_socks[i];
            }
        }
        int ret = select(maxfd + 1, &readset, NULL, NULL, NULL);
        if(ret < 0){
            if(errno == EINTR){
                continue;
            }else{
                printf("select error! %s
", strerror(errno));
                exit(0);
            }
        }

        if(ret > 0){
            for(int i=0; i<MAX_PORTS; i++){
                if(!FD_ISSET(server_socks[i], &readset)){
                    continue;
                }
                socklen_t addrlen = sizeof(addr);
                int sock = accept(server_socks[i], (struct sockaddr *)&addr, &addrlen);
                if(sock == -1){
                    goto sock_err;
                }
                connections ++;
                printf("connections: %d, fd: %d
", connections, sock);
            }
        }
    }

    return 0;
sock_err:
    printf("error: %s
", strerror(errno));
    return 0;
}

Test program (client)

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <arpa/inet.h>
#include <netinet/tcp.h>

int main(int argc, char **argv){
    if(argc <=  2){
        printf("Usage: %s ip port
", argv[0]);
        exit(0);
    }

    struct sockaddr_in addr;
    const char *ip = argv[1];
    int base_port = atoi(argv[2]);
    int connections = 0;

    bzero(&addr, sizeof(addr));
    addr.sin_family = AF_INET;
    inet_pton(AF_INET, ip, &addr.sin_addr);

    int index = 0;
    while(1){
        if(++index >= 10){
            index = 0;
        }
        int port = base_port + index;
        printf("connect to %s:%d
", ip, port);
        addr.sin_port = htons((short)port);

        int sock;
        if((sock = socket(AF_INET, SOCK_STREAM, 0)) == -1){
            goto sock_err;
        }
        if(connect(sock, (struct sockaddr *)&addr, sizeof(addr)) == -1){
            goto sock_err;
        }

        connections ++;
        printf("connections: %d, fd: %d
", connections, sock);

        if(connections % 10000 == 9999){
            printf("press Enter to continue: ");
            getchar();
        }
        usleep(1 * 1000);
    }

    return 0;
sock_err:
    printf("error: %s
", strerror(errno));
    return 0;
}

Testing 100 k idle connections showed the user‑space process used less than 1 MB, while the kernel consumed about 200 MB; extrapolating to a million connections suggests roughly 2 GB of kernel memory (≈2 KB per connection).

Memory can be tuned via:

/proc/sys/net/ipv4/tcp_wmem
/proc/sys/net/ipv4/tcp_rmem

These files control TCP send and receive buffer sizes.

3. How much memory does the application itself need?

Measurements indicate the user‑space program adds negligible memory overhead; the kernel dominates the consumption.

4. Does throughput exceed network limits?

If 20 % of a million connections are active, each sending 1 KB/s, the required bandwidth is about 1.6 Gbps, so a 10 Gbps (10 Gigabit) NIC is recommended.

Conclusion

Linux must have its kernel parameters and limits raised to support C1000K. A realistic server should have at least 2 GB of RAM for the kernel, 10 GB overall if the application also needs memory, and a 10 Gbps network interface.

These figures are theoretical; real‑world workloads will need additional CPU and memory resources.

References:

http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

http://www.lognormal.com/blog/2012/09/27/linux-tcpip-tuning/