How to Reach 900k Requests/sec with Nginx: Full Server & TCP Tuning Guide

This article is part 2 of a series on building a high‑performance server cluster capable of handling millions of requests per second; it focuses on configuring Nginx for extreme throughput and further boosting network bandwidth by tuning the Linux TCP stack.

Install and basic Nginx configuration

Install Nginx: yum -y install nginx Backup the original configuration and edit /etc/nginx/nginx.conf:

cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.orig

vim /etc/nginx/nginx.conf

Key directives (adjust values to match your hardware): worker_processes 24; – maximum number of CPU cores. worker_rlimit_nofile 200000; – raise the limit of open file descriptors.

error_log /var/log/nginx/error.log crit;

worker_connections 4000;

– together with worker_processes defines max clients.

use epoll;

multi_accept on;

open_file_cache max=200000 inactive=20s;

open_file_cache_valid 30s;

open_file_cache_min_uses 2;

open_file_cache_errors on;

access_log off;

sendfile on;

tcp_nopush on;

tcp_nodelay on;

keepalive_timeout 30;

keepalive_requests 100000;

reset_timedout_connection on;

client_body_timeout 10;

send_timeout 2;

gzip on;

and related gzip settings.

Start Nginx and enable autostart

service nginx start

chkconfig nginx on

Load testing with Tsung

Edit the Tsung configuration (e.g., ~/.tsung/tsung.xml) to point to your web server and then run: tsung start After the test finishes, press Ctrl+C and use the previously defined alias treport to view the report.

TCP stack tuning (applicable to any web server)

Install benchmarking tools: yum -y install netperf iperf Edit /etc/sysctl.conf and add the following parameters (adjust as needed):

# Increase local port range
net.ipv4.ip_local_port_range = 2000 65000
net.ipv4.tcp_window_scaling = 1

# Backlog and listen queue
net.ipv4.tcp_max_syn_backlog = 3240000
net.core.somaxconn = 3240000
net.ipv4.tcp_max_tw_buckets = 1440000

# TCP buffer sizes
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_congestion_control = cubic

Apply the new settings: sysctl -p /etc/sysctl.conf After each change, run a network benchmark (e.g., with netperf or iperf) to measure its impact.

Following these Nginx and kernel adjustments, the author observed a stable 904 k requests per second on a 24‑core machine, up from about 560 k req/s before the tuning.