How Nginx Handles 100K Concurrent Connections: Inside Its Event-Driven Architecture
This article dissects Nginx's high-concurrency architecture, explaining its event-driven model with epoll, master-worker process design, thundering herd solutions, zero-copy sendfile, request processing phases, and capacity planning formulas, revealing why configuration must align with OS resource limits.
Three Production Incidents That Expose the Knowledge Gap
The article opens with three real-world incidents:
Incident 1: During a flash sale, Nginx gateway logs flooded with "too many open files" errors, returning 502s. Adding more Nginx instances didn't help. Root cause: the OS file descriptor limit (ulimit) was 1024, while worker_connections was set to 50000. Nginx cannot open more connections than the OS allows.
Incident 2: A 32-core machine ran worker_processes 32, but QPS dropped 30% compared to 8 workers. Flame graphs revealed the classic thundering herd problem: all 32 workers woke up on each new connection, only one succeeded, wasting CPU on context switches. Nginx provides accept_mutex and SO_REUSEPORT to solve this; ignorance of these mechanisms leads to self-inflicted performance regressions.
Incident 3: Replacing Apache with Nginx didn't improve throughput. The bottleneck was not the gateway process model but the forwarding method: Apache's mod_proxy held a backend connection per request, exhausting the backend connection pool under long-lived connections. Nginx merely moved the queue entrance; the queue itself remained unchanged.
These incidents share a theme: Nginx performance is not magic but a complete design from OS event model to process cooperation mechanisms. Understanding it turns configuration from lottery into precision.
Technical Analysis: Two Pillars — Process Model & Event Model
1.1 Concurrency Model Fork in the Road
Web server concurrency models boil down to "resources per connection":
Process model (Apache prefork) — one process per connection; best isolation but memory explodes; caps at thousands.
Thread model (Apache worker) — one thread per connection; better memory but thread scheduling overhead grows linearly; struggles at tens of thousands.
Event-driven model (Nginx/Node/Netty) — a few resident workers manage tens of thousands of connections via an event loop. A connection is just an event source plus a small state struct, not an independent execution flow. 10,000 connections need only a handful of worker processes plus 10,000 lightweight connection structs, not 10,000 processes (each tens of MB).
Nginx handles 100K concurrency not because its code is exceptionally fast, but because it doesn't hire a "waiter" per connection; it hires a few waiters watching a whole wall of service bells.
1.2 epoll: The Engine of Event-Driven
The core challenge: efficiently knowing which connections have data ready. Three generations:
select — copies entire fd set into kernel, then scans all fds for readiness; 1024 fd limit; O(n) copy + scan is disastrous at scale.
poll — removes 1024 limit but retains O(n) scan.
epoll — three calls change everything: epoll_create builds an event table; epoll_ctl registers fds once for reuse; epoll_wait returns only ready fds — O(1) wait, O(ready) return. With 10,000 connections and only 50 ready, epoll_wait processes just 50.
This is the "service bell" implementation: each connection hangs a bell; only when it rings does a waiter approach; no one lifts every pot lid.
Two critical details:
LT (level-triggered) vs ET (edge-triggered) — LT reports continuously while data remains in the read buffer (simple, hard to miss events); ET reports once on state change (must read until EAGAIN, else event lost). Nginx defaults to LT for stability; ET requires non-blocking I/O.
epoll only solves "waiting", not "doing" — after data is ready, HTTP parsing, response assembly, upstream forwarding are the worker's own work. CPU-intensive tasks (e.g., complex Lua) block all connections on that worker, foreshadowing OpenResty pitfalls.
1.3 Master-Worker: Why Multi-Process Not Multi-Thread
Nginx adopts a hybrid: multi-process + single-process event-driven .
Master process acts as foreman: reads config, binds ports, forks workers, monitors worker health, handles signals (reload = master forks new workers, signals old workers to gracefully exit).
Worker processes do all the heavy lifting: accept connections, run event loop, process requests.
Three reasons this beats multi-threading:
Isolation — a worker crash (C segfault) doesn't affect others; master spawns a replacement; blast radius confined to one process.
Lock-free — each worker independently handles its own connection set; no shared mutable state (variables live in connection/request structs); no lock contention or deadlocks.
Hot reload — during reload, old and new workers coexist; old workers finish in-flight requests before exiting; connections stay up, service uninterrupted. Thread pools struggle to migrate in-flight connections cheaply.
Classic worker count formula: equal to CPU cores ( worker_processes auto auto-detects). Each worker is an event pump that saturates one core; more workers increase thundering herd and context switches (Incident 2), fewer waste cores. Exception: disk I/O intensive workloads (large static file serving) — workers can exceed core count because they yield while waiting for disk, allowing multiple workers to interleave and keep CPU busy.
1.4 Thundering Herd and Its Three Solutions
Incident 2's thundering herd deserves a dedicated section as the #1 source of "config correct but performance collapsed". Three solutions in historical order:
accept_mutex (mutex lock) — Nginx traditional approach: only one worker holds the lock to accept; others wait. Eliminates herd but adds lock overhead and "lock latency" (connection distribution less even).
SO_REUSEPORT — kernel 3.9+: multiple sockets bind same port; kernel load-balances connections across sockets ; no lock, more even distribution. Nginx 1.9.1+ enables with listen reuseport.
EPOLLEXCLUSIVE — kernel 4.5+ compromise: wakes only one waiter instead of all.
Modern default answer is reuseport : significant gains in short-connection high-concurrency scenarios (e.g., LVS-facing ingress layer); less difference for long-connection workloads. Incident 2's root cause had a hidden factor: 32 workers + accept_mutex off + kernel without reuseport support — three conditions combined to unleash full thundering herd.
1.5 One Request's Journey: 13 Phases
Nginx slices HTTP request processing into 11 standard phases (plus 2 content-handling entry points), fixed order:
SERVER_REWRITE → FIND_CONFIG → REWRITE → POST_REWRITE → PREACCESS → ACCESS → POST_ACCESS → PRECONTENT → CONTENT → LOGValue of this map: every directive maps to a phase . rewrite module directives run in REWRITE phase; limit_req / limit_conn / auth_basic in ACCESS; try_files / proxy_pass in CONTENT. Directives in the same phase execute in module order; cross-phase order is strictly by phase sequence. This explains countless "why my limit_req doesn't block" issues (e.g., rewrite phase changes URI, then access phase matches limit config against new URI). Later chapters (location/rewrite/rate-limiting) will revisit this map.
1.6 Nginx's Place in the Gateway Ecosystem
Positioning for this series:
Nginx handles L4/L7 ingress — connection management, load balancing, static serving, caching, rate limiting, TLS termination.
It does NOT do business routing — semantic routing, auth, canary releases belong to "programmable gateways" like Spring Cloud Gateway / OpenResty.
Typical layering:
SLB/LVS → Nginx ingress cluster (rate limit + static + cache + TLS) → Business gateway (routing/auth/canary) → Microservices. Nginx and business gateway are complementary, not replacements: former pursues extreme throughput and stability (C, decade-stable kernel); latter pursues business flexibility (Java/Lua). Series roadmap: 02 location & rewrite → 03 reverse proxy & load balancing → 04 caching & static → 05 rate limiting → 06 HTTPS & kernel tuning → 07 logging & debugging → 08 HA with Keepalived → 09 OpenResty dynamic gateway → 10 production finale.
Core Flow: From Connection Establishment to Response Return
2.1 Complete Timeline of an HTTP Request
Using a static request as example:
T1 Client TCP three-way handshake; kernel completes handshake (SYN queue / accept queue); worker's epoll_wait receives listen fd readable event.
T2 Worker accept gets new connection fd, registers it with epoll, allocates connection struct.
T3 Receives HTTP request headers (possibly across multiple TCP packets); state machine parses incrementally; once complete, enters 11-phase pipeline.
T4 Matches server/location; per CONTENT phase config (static file → open; proxy → upstream).
T5 Static file: open file → sendfile zero-copy send (data from page cache straight to NIC, bypassing user space).
T6 Response headers + body written back; under keepalive connection returns to event loop for next request; on timeout or close, released.
T3's parsing state machine and T5's sendfile are two more sources of "fast": one turns protocol parsing into O(header size) incremental state machine; the other reduces data copying from four copies to zero.
2.2 Master-Worker Collaboration Details
Master-worker communication uses command pipe + signals . Reload flow walked end-to-end:
Master receives HUP, re-reads config and does syntax check (failure keeps old config running — why bad config won't crash service).
Forks new workers (with new config).
Sends QUIT to old workers — old workers stop accepting new connections, finish in-flight connections then exit (long connections have worker_shutdown_timeout safety net).
Old processes all gone; reload complete.
This mechanism explains two production phenomena: seeing old and new workers coexist after reload is normal; kill -9 on a worker triggers master to immediately respawn it (deleting process is inferior to signals). If a worker crashes repeatedly (multiple times per second), master enters exponential backoff — log shows spawn intervals growing longer; that's a C-level bug causing crash loops.
2.3 Connection Limits: The Complete Chain
Expanding Incident 1's "too many open files" into the full limit chain — every layer can be the bottleneck:
Per-process fd limit — ulimit -n (soft/hard); Nginx config worker_rlimit_nofile lets master raise it, but hard limit requires systemd LimitNOFILE change.
worker_connections — per-worker connection ceiling (includes upstream connections! Proxy scenario: each dynamic request consumes 2 fds — client 1 + upstream 1 — so actual capacity = workers × worker_connections ÷ 2).
System globals — fs.file-max, net.core.somaxconn (listen queue), ip_conntrack (stateful firewall connection tracking table, often explodes in LB scenarios).
Port resources — Nginx as client connecting upstream uses source ports (approx 60K via net.ipv4.ip_local_port_range); TIME_WAIT buildup exhausts ports; tcp_tw_reuse mitigates.
Capacity planning essence: walk this chain layer by layer; the shortest board determines real limit.
2.4 Keepalive and the Economics of Request Reuse
HTTP/1.0 required TCP handshake (3 packets) + teardown (4 packets) per request; 1000 QPS = 7000 pure protocol packets. Keepalive reduces protocol overhead to near zero.
Nginx has two separate keepalive segments to configure:
Client side — keepalive_timeout (default 75s) and keepalive_requests (default 1000, 100 before 1.19.10). Too long piles up idle connections consuming fds; too short lowers reuse rate. Rule of thumb: 30–65s.
Upstream side — upstream { keepalive N; } — this is Incident 3's real cure : enables long-lived connection pool to backend; without it, every request creates a new connection (backend TIME_WAIT pileup, handshake overhead).
Hidden trap: upstream keepalive requires three pieces together — proxy_http_version 1.1 and clearing Connection header ( proxy_set_header Connection "";). HTTP/1.0 defaults to no keepalive, and proxy forwards client's Connection: close verbatim. Missing any piece renders the keepalive pool useless.
Core Source Code: Five Key Logic Sections
3.1 Worker Main Loop: Event Pump Skeleton
// ngx_worker_process_cycle simplified
for ( ;; ) {
// 1. Handle timers: expired event callbacks
ngx_process_events_and_timers(cycle);
// Internal: epoll_wait for events
// -> dispatch each to its handler
// -> process expired timers (timeout/keepalive)
if (ngx_terminate) {
ngx_worker_process_exit(cycle);
}
if (ngx_quit) {
// graceful reload exit:
// close listening sockets (stop accept)
// wait for in-flight connections to finish then exit
if (!ngx_exiting) {
ngx_close_listening_sockets(cycle);
ngx_exiting = 1;
}
}
}3.2 accept_mutex: Thundering Herd Lock Distribution Logic
// ngx_event_accept attached to listen fd
// Event entry tries lock first (when accept_mutex enabled)
if (ngx_accept_disabled > 0) {
// I'm too busy (connections exceed 7/8 of average)
// Yield lock, don't participate in accept
ngx_accept_disabled--;
return;
}
if (ngx_trylock_accept_mutex(cycle)
== NGX_ERROR) {
return; // didn't get lock, skip accept this round
}
// Lock-holding worker puts listen fd into
// epoll interest set -> accept new connection
// After handling, releases lock for next worker
// disabled counter = load self-balancing valveTwo clever designs hidden here:
Load self-balancing — when a worker's connection count exceeds 7/8 of the average, it voluntarily yields the lock (busy workers take less work). This is smarter than a bare lock.
Lock only covers the accept instant — worker grabs lock, adds listen fd to epoll, processes, releases immediately; lock hold time is minimal.
But note: since 1.11.3 accept_mutex defaults to off — because EPOLLEXCLUSIVE and reuseport solve the same problem at kernel level more efficiently. Don't blindly carry accept_mutex on from old configs.
3.3 11-Phase Pipeline: Checker-Driven
// ngx_http_core_run_phases simplified
void ngx_http_core_run_phases(r) {
ph = cmcf->phase_engine.handlers;
// phase_engine is compile-time flattening of 11
// phases' handlers into a single array
while (ph[r->phase_handler].checker) {
rc = ph[r->phase_handler].checker(r,
ph[r->phase_handler].handler);
if (rc == NGX_OK) return; // suspend
// NGX_DECLINED -> next handler
// NGX_AGAIN/DONE -> async wait for event
// NGX_DONE/forbidden -> terminate
}
}
// Each request has a phase_handler cursor
// Async (e.g., upstream) resumes where left off3.4 sendfile: Zero-Copy Call Chain
// Traditional read+write: 4 copies, 4 context switches
// 1. read: disk -> kernel page cache (DMA)
// 2. read: page cache -> user space (CPU copy)
// 3. write: user space -> socket buffer (CPU)
// 4. write: socket -> NIC (DMA)
// sendfile: 2 copies, 0 user-space round trips
// 1. disk -> page cache (DMA)
// 2. page cache -> NIC (DMA with gather)
// Only passes file descriptor + offset; data never
// touches user space -> worker not interrupted
# nginx.conf equivalents:
sendfile on; # enable zero-copy
tcp_nopush on; # wait for packet to fill before send
tcp_nodelay on;# send small packets immediately (keepalive)3.5 Config Validation: Why Bad Config Won't Crash Service
# nginx -t execution path
# 1. Parse config tree, invoke each directive's
# set callback for semantic validation
# 2. Attempt to open all referenced files/ports
# 3. Any failure -> non-zero exit
# 4. Reload: master runs same validation first
# Failure aborts reload, old config keeps serving
# -> built-in rollback protection
# Production discipline:
# nginx -t && nginx -s reload
# NEVER reload -f skipping validation
# CI: add config lint (nginx -t -c)These five pieces form Nginx's skeleton: a worker is an event pump + phase pipeline combo; master uses validate-then-switch for atomic config changes;
accept_mutex</strong> uses <strong>yield counter</strong> for load self-balancing; <code>sendfile</strong> uses <strong>user-space bypass</strong> to push I/O to the limit. <strong>Reading source isn't to memorize function names, but to build causal links between config and behavior</strong> — knowing <code>accept_disabledexists explains why worker connection counts need headroom.
Advanced Topics: Four Accounts You Must Calculate Upfront
4.1 Capacity Account: How Many Connections Can Your Nginx Really Handle
Turn the 2.3 chain into a formula with real numbers. Pure reverse proxy scenario:
Theoretical capacity = workers × worker_connections ÷ 2 (one in, one out). 8 workers × 50000 ÷ 2 = 200K connections — provided all four system layers are fully opened : worker_rlimit_nofile ≥ 100000, systemd LimitNOFILE raised, somaxconn ≥ 65535, conntrack table sized. Check each layer: ulimit -n (current), ss -lnt Recv-Q/Send-Q, dmesg for conntrack full, netstat TIME_WAIT count. Incident 1's lesson becomes a mandatory pre-flight check: First thing on a new Nginx box is verify the four system limits, not edit server blocks. Connection count is just the entrance; the real bottleneck often hides in the shortest board of the 2.3 chain.
4.2 CPU Account: Worker Count and Benchmark Curves
Worker count isn't guesswork; use benchmark curves:
Start at half core count, step up to core count, then 2× core count; record QPS and P99.
Pure proxy/static workloads' sweet spot almost always at "core count" (Incident 2: 32 cores with 32 workers plus thundering herd actually degraded).
Disk I/O intensive (large file downloads) may peak at 2–4× core count.
Watch two secondary metrics: context switch rate ( vmstat cs column — spikes with thundering herd/over-parallelism) and softirq share (si column — NIC interrupt handling). Often overlooked: worker_cpu_affinity — pinning workers to cores improves cache hit rate, 5–10% gain on NUMA boxes, but verify NIC interrupt affinity first; don't let all NIC interrupts pile on CPU0 while workers wait on other cores.
4.3 Memory Account: How Much RAM for 100K Connections
Connection memory cost in three parts:
Connection struct (~few hundred bytes)
Request struct + buffers (default client_header_buffer 1k + possible large header buffer 8k; proxy adds upstream buffers)
Kernel socket buffers ( tcp_rmem / tcp_wmem dynamic; idle connections negligible, active high-throughput connections can reach hundreds of KB)
Rough math: 100K idle keepalive connections ~100–200MB; 100K active proxy connections could reach several GB. Conclusion: memory is never Nginx's bottleneck for high concurrency; fds and CPU are — that's the event-driven model's point. Real memory spike to guard against is "buffer bloat": with proxy_buffering on, slow clients hog upstream buffers (covered in chapter 03 with proxy_max_temp_file_size and buffer tuning).
4.4 Smoothness Account: Reload, Upgrade, Rollback
Three tiers of production change smoothness:
Reload (HUP) — config change; old and new workers coexist; zero-downtime in seconds.
Binary upgrade (USR2+WINCH) — old and new masters coexist: send USR2 to start new master (new binary, new pid file); then WINCH to gracefully drain old master's workers; after verification, send QUIT to old master to finish. Rare mechanism allowing binary-level smooth rollback — if issues, send HUP to old master for one-click rollback.
Config rollback — reload is naturally atomic (validation failure auto-aborts), but "semantically wrong but syntactically correct config" (e.g., rate limit threshold set 10x too low) has no safety net; only defense is config versioning (git-managed config + CI nginx -t + change approval).
Three disciplines: config always in git with CI validation; binary upgrades via USR2 flow, never stop/start; after every change, diff live nginx -T against git baseline.
Learning Path: From Configuring to Understanding Principles
Week 1: Build lab environment — Docker Nginx, four controlled experiments: worker count 1/4/8 benchmark curves; accept_mutex on/off context switch comparison; upstream keepalive three-piece TIME_WAIT comparison; sendfile on/off CPU consumption comparison. One conclusion note per experiment.
Weeks 2–3: Script the four-layer resource limit checks (ulimit/somaxconn/conntrack/ports), run full health check on existing Nginx fleet, produce first capacity account report.
Weeks 4–5: Deep-read three source files — ngx_worker_process_cycle (event loop skeleton), ngx_event_accept (thundering herd lock), ngx_http_core_run_phases (phase pipeline); combine with strace to observe real syscall sequences ( epoll_wait / accept4 / sendfile).
Advanced: Comparative reading — Apache event MPM vs Nginx worker architecture differences; Envoy vs Nginx thread model differences; understand "why Service Mesh era Envoy chose multi-threading while Nginx sticks to multi-process" and the trade-offs on each side.
High-Frequency Interview Questions (with Summary Answers)
The article includes 20 detailed Q&A pairs covering: why Nginx handles high concurrency; epoll vs select/poll and LT vs ET; why multi-process not multi-thread; worker_processes tuning and thundering herd; thundering herd solutions; reload zero-downtime mechanics; worker_connections meaning and division by 2; systematic "too many open files" resolution; two keepalive segments and upstream keepalive pitfalls; sendfile speed and tcp_nopush/tcp_nodelay interplay; 11 request phases importance; capacity planning system limits; TIME_WAIT mitigation; Nginx vs business gateway division; nginx -t importance and reload rollback; master/worker crash behavior; binary upgrade/rollback; request latency attribution; idle keepalive mitigation; zero-to-hero machine performance evaluation.
One-Page Cheat Sheet
Two pillars: Event-driven (epoll returns only ready, no full scan) + master-worker multi-process (lock-free, isolation, hot reload).
Three mantras: Connection ≠ execution flow (event model saves "waiters"); config effectiveness sits on system resources (fd/conntrack/ports all four layers must open); upstream keepalive three-piece missing one = decoration (keepalive N + http_version 1.1 + clear Connection header).
Thundering herd three fixes: accept_mutex (traditional lock + yield self-balancing), reuseport (kernel distributes, modern default), EPOLLEXCLUSIVE (kernel wakes one).
Reload two insurances: Validation failure auto-abort (syntax rollback protection); old workers finish in-flight then exit.
Capacity formula: Theoretical = workers × worker_connections ÷ 2; shortest system board = real ceiling.
One-sentence summary: Nginx's speed isn't magic — it changed "hire a waiter per connection" to "a few waiters watch a whole wall of service bells"; understanding this model gives every config item a cause.
Summary & Next Chapter Preview
This chapter answered "Why Nginx is fast": event-driven model makes connections not consume execution flows; master-worker multi-process brings lock-free, isolation, hot reload; epoll/sendfile/thundering-herd governance push every link's overhead to the limit; simultaneously established the complete capacity planning chain — worker_connections is just config layer, system four-layer limits are the real ceiling. Next chapter enters the most common battlefield: location matching rules and rewrite module — eight iron laws of priority, three semantics of last/break/redirect, ten common "location not working" death traps.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
