How to Scale Nginx for 2 Million Long‑Lived Connections: A Practical Guide

Background and Goal

The objective is to evaluate how many simultaneous long‑lived (hold) connections a server can sustain, a scenario common in comet‑style push systems such as chat or instant‑messaging services, where connections remain idle until data is generated.

Resource Analysis

Long connections consume mainly memory, while CPU and network are largely idle. The hypothesis is that sufficient RAM alone determines the maximum connection count, but kernel data structures and limits must also be considered.

Server Preparation

A high‑memory machine (Dell R710, 2 × Xeon E5520, ~24 GB RAM) runs Nginx with a custom comet module that accepts a request and holds the connection without responding. Nginx’s status module is used to monitor the maximum connections.

Key sysctl parameters are tuned in /etc/sysctl.conf:

net.core.somaxconn = 2048
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 4096 16777216
net.ipv4.tcp_wmem = 4096 4096 16777216
net.ipv4.tcp_mem = 786432 2097152 3145728
net.ipv4.tcp_max_syn_backlog = 16384
net.core.netdev_max_backlog = 20000
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_orphans = 131072

These settings reduce per‑socket memory usage (by setting default buffers to 4 KB) and raise limits for pending connections and orphaned sockets.

File Descriptor Limits

To hold millions of sockets, the maximum number of open file descriptors must be increased. On the client side a limit of 100 000 is set, while the server aims for 2 000 000.

On kernels prior to 2.6.25 the hard limit is capped at 1 048 576 (1 M). Upgrading to kernel 2.6.32 allows raising the limit via /proc/sys/fs/nr_open: echo 2000000 > /proc/sys/fs/nr_open After the upgrade, /etc/security/limits.conf is updated:

admin    soft    nofile  2000000
admin    hard    nofile  2000000

Client Preparation

Each client machine can open at most ~64 k ports (1024‑65535). To reach 2 M connections, about 34 client machines (or virtual IPs) are required. The client program, built with libevent, continuously creates new connections to the server.

The local port range is expanded:

net.ipv4.ip_local_port_range = 1024 65535

Final Tuning and Results

Additional tweaks include reducing Nginx’s request_pool_size from the default 4 KB to 1 KB and aligning tcp_wmem and tcp_rmem defaults to 4 KB.

After these adjustments, the test achieved 2 000 000 concurrent long connections. Nginx’s status page showed the connection count, and memory usage remained within the server’s RAM capacity, as illustrated by the screenshots below.