Nginx Production Mastery: Capacity Planning, Safe Releases, Incident Grading & 10 Postmortem Templates
This comprehensive guide distills Nginx production hardening into four critical accounts—capacity, stability, observability, and change management—backed by three real incidents, a complete nginx.conf skeleton, health-check and failover scripts, a phased release process, ten incident categories, source-code analysis of connection limits, retries, reloads, and logging, plus SLO-driven drills and 20 interview Q&As.
Introduction
The article concludes a 10-part Nginx series by shifting from individual features (routing, caching, rate-limiting, etc.) to a production-grade system that survives traffic spikes, audits, and handovers. The author argues that the same nginx.conf can run for three years without incidents or crash within a week—the difference lies in four "accounts" that must be calculated: capacity, stability, observability, and change management. Two concrete assets are provided: a full production nginx.conf skeleton and a pair of battle-tested health-check and failover scripts.
Three Incidents, Three Lessons
Incident 1 – One-line change, 40-minute 502 outage: An engineer increased client_max_body_size on only one of two servers, skipped validation and canary, and three days later a scale event shifted traffic to the mismatched node, causing 413 errors that saturated connection pools.
Incident 2 – Connections exhausted three minutes into a promotion: Load testing showed 40% CPU at target QPS, but real traffic had 10-second upstream timeouts that held connections. worker_connections (20,480) was hit while ulimit -n remained at 1,024, so log file descriptors couldn't be opened. Two miscalculations: tail latency amplifies concurrency far beyond QPS × avg RT, and connection slots and file descriptors are separate limits.
Incident 3 – Half-hour outage, no clear impact scope: Multiple engineers simultaneously reloaded, restarted upstreams, and changed timeouts without coordination, external communication, or a change log. Recovery was lucky; root cause remained unknown.
Key takeaway: during incidents the most expensive resource is order, not technology —without severity grading, a single commander, and a timeline, recovery relies on luck and lessons are lost.
Four Production Accounts
1. Capacity Account: From Peak QPS to Machine Count
Capacity is a calculable chain, not a feeling. Start with Little's Law: concurrent connections ≈ peak QPS × average RT (seconds) . Example: 20,000 QPS × 50 ms = 1,000 steady-state connections. But tail latency is the killer —if 1% of requests take 10 seconds, average RT jumps to 150 ms and concurrency triples. Therefore capacity must be sized on P99 (or avg RT × 3), not average. Apply a 1.3 safety factor, then add upstream keepalive pools and internal sub-requests to get the true worker_connections.
Three hard ceilings, any one of which becomes the bottleneck:
Connection slots – worker_connections per worker.
File descriptors – each connection consumes an fd; plus log files, listen sockets, upstream connections. ulimit -n must be at least 2× worker_connections (Incident 2 adjusted only the former).
Memory – ~8–16 KB per connection (read/write buffers + structs). 20,000 connections = 200–300 MB, plus shared dictionaries and caches.
2. Stability Account: Five Parameters That Decide Life or Death
Three-part timeouts: connect 1 s (intra-DC >1 s is abnormal), send 10 s, read 10 s. Too long → slow upstream eats slots (Incident 2); too short → legitimate long tasks killed.
Keepalive staggering: Nginx keepalive_timeout 65s, upstream 60 s – let upstream close first . Reverse order causes Nginx to reuse a connection the peer already closed, yielding sporadic 502s that are extremely hard to reproduce.
Buffering: proxy_buffering on, proxy_buffers 8 4k, busy_buffers_size 8k – prevents slow clients from occupying upstream connections.
Retry budget: tries 2, total 3 s; retry only on error, not timeout (timeout may mean upstream already processed – retrying a write causes duplicate orders).
Request body limits: client_max_body_size with client_body_buffer_size to prevent large payloads from filling disk.
3. Observability Account: Four Golden Signals & Three Blind Spots
Golden signals: Traffic (QPS, bandwidth), Errors (5xx rate, 499 rate), Latency (P50/P95/P99), Saturation (connections/limit, CPU, file descriptors).
Three blind spots:
Logging only $request_time without $upstream_response_time – the former includes client network time, making it impossible to distinguish slow clients from slow backends. Both are needed to compute "gateway overhead."
Aggregating error rate cluster-wide – a 1% cluster error rate may come from a single node at 100%. Must aggregate by $upstream_addr.
Treating 499 as success – 499 means client closed early, usually because gateway/backend was too slow. It reflects user experience better than 5xx.
4. Change Account: Five-Step Release & 60-Second Rollback
Incident 1's root cause was not the config line but the absence of a change process. Five steps:
Config as code – all .conf in Git, versioned directories per release, symlink points to current version, no manual edits .
Post-render validation – template + variables → final config; nginx -t must run on a temp instance matching production version.
Canary one node – reload one server, watch four golden signals for 5 minutes, check error.log for new warnings.
Phased rollout – 20% → 50% → 100%, each batch with an observation window; never full blast .
Rollback in 60 seconds – symlink back + reload; rollback script pre-written and rehearsed. Team maturity = how many minutes to rollback.
5. Fault Account: Severity Grading & Five-Step Response
Grading first assigns ownership: P0 = site down or revenue loss (10-min call, all hands); P1 = core feature degraded or >10% users affected; P2 = non-core, partial users; P3 = no user impact.
Five steps: Stop bleeding (rollback, drain, degrade – restore first, find root cause later) → Locate (only three signals: new error.log entries, upstream error rate, last change) → Recover (confirm metrics back to baseline, not "looks ok") → Postmortem (within 24 hrs, 5 Whys to mechanism level) → Improve (each action has owner + deadline). Only one incident commander allowed ; everyone else executes or observes – arguing while changing is the top cause of secondary escalation.
6. Cost Account: Density vs. Redundancy Balance
Single-node capacity ≤ 30% of cluster – higher trades stability for cost.
Redundancy starts at N+1 (survive one node loss without capacity drop); critical paths consider cross-AZ N+2.
Biggest cost often isn't servers but egress bandwidth – push static assets/images to CDN/object storage, gateway returns only HTML/JSON, saves ~70% egress.
Hidden bill: complexity – every extra LVS layer or dynamic gateway adds a failure point and ops burden; use cloud LB if it solves the problem.
Two Complete Production Assets
2.1 Full Production nginx.conf Skeleton
# /etc/nginx/nginx.conf production skeleton
user nginx;
worker_processes auto;
worker_rlimit_nofile 65535;
pid /run/nginx.pid;
events {
worker_connections 20480;
use epoll;
multi_accept on;
accept_mutex off; // disable on multi-core
}
http {
include mime.types;
default_type application/json;
server_tokens off; // hide version
sendfile on;
tcp_nopush on;
keepalive_timeout 65;
client_max_body_size 20m;
client_body_buffer_size 128k;
log_format main '$remote_addr $status '
'$rt $upstream_addr';
access_log /var/log/nginx/acc.log
main buffer=64k flush=2s;
error_log /var/log/nginx/err.log warn;
gzip on;
gzip_min_length 1k;
gzip_comp_level 5;
gzip_types text/css text/plain
application/javascript;
proxy_connect_timeout 1s;
proxy_send_timeout 10s;
proxy_read_timeout 10s;
proxy_next_upstream error timeout;
proxy_next_upstream_tries 2;
proxy_next_upstream_timeout 3s;
upstream api_svc {
server 10.0.1.11:8080
max_fails=3 fail_timeout=10s;
server 10.0.1.12:8080
max_fails=3 fail_timeout=10s;
keepalive 128;
}
include conf.d/*.conf;
include /data/release/cur/*.conf;
}Every line answers an incident: worker_rlimit_nofile 65535 – aligns Nginx's fd ceiling with system ulimit and limits.conf (Incident 2). accept_mutex off – reduces thundering herd on high concurrency; enable only for low concurrency.
Dual include – conf.d for platform-level (few changes), /data/release/cur is a symlink to current business version; rollback = flip symlink + reload . log_format emits both $rt and $upstream_addr – the only way to compute per-node error rates. access_log buffer=64k flush=2s – batches writes, cuts syscalls from per-request to per-second; trade-off: last 2 seconds lost on crash (covered by error.log).
2.2 Health-Check & Failover Scripts
Health Check ( /usr/local/bin/ngx_check.sh )
#!/bin/bash
# /usr/local/bin/ngx_check.sh health check
echo "== 1 Config syntax =="
nginx -t 2>&1 | tail -1
echo "== 2 FD limits =="
ulimit -n; cat /proc/sys/fs/file-max
echo "== 3 Connection slots =="
curl -s 127.0.0.1/status
| grep -E 'Active|reading'
echo "== 4 Upstream health =="
for ip in 10.0.1.11 10.0.1.12;
do
c=$(curl -s -o /dev/null -m 2 \
-w '%{http_code}' \
http://$ip:8080/health)
echo "$ip -> $c"
done
echo "== 5 Error log TOP =="
tail -2000 /var/log/nginx/err.log \
| awk '{print $3}' \
| sort | uniq -c | sort -rn | headFive steps map to five failure classes: syntax error (reload fails but old process runs – most insidious), FD exhaustion (Incident 2), connection slot exhaustion (Active near worker_connections × workers), upstream single-node failure, error log spike. Must complete in ≤10 seconds or on-call won't use it; absolute paths because cron/patrol PATH often lacks /usr/sbin.
Failover ( /usr/local/bin/ngx_failover.sh )
#!/bin/bash
# /usr/local/bin/ngx_failover.sh drain
set -e
PEER=10.0.0.12
# 1 Drain upstream first, stop new traffic
curl -s -X POST http://gw/offline \
-d "ip=$(hostname -i)" || true
# 2 Graceful quit: stop accepting new connections
nginx -s quit
for i in $(seq 1 30); do
pgrep -x nginx >/dev/null || break
sleep 1
done
# 3 Force kill after timeout – better hard down than half-alive
pgrep -x nginx >/dev/null \
&& pkill -9 nginx
echo "done, traffic -> $PEER"Order is critical: drain traffic before stopping process – reverse causes "process stopped but upstream still sends requests" → 502s for tens of seconds. nginx -s quit is graceful (finishes existing requests), but long-lived connections (WebSocket, SSE) may never end, hence 30 s timeout + SIGKILL fallback. Script must be a button in the release platform and rehearsed quarterly – an un-rehearsed script's first run is always during a real incident.
Release Timeline: Commit to Full Rollout
T0 Commit: change templates, not artifacts; PR must include "impact scope" and "rollback method" columns.
T1 Build: render full config, create timestamped + git-sha release dir; run static checks (duplicate server_name, missing upstream refs, missing cert paths).
T2 Pre-check: nginx -t on staging + tcpcopy or traffic replay validation.
T3 Canary: reload first node, observe 5 minutes.
T4 Ramp: 20% → 50% → 100%, 5-minute window each batch.
T5 Wrap-up: snapshot metrics for this batch – enables comparison during incidents.
The key isn't automation depth but explicit go/no-go criteria per batch – error rate, P99, 499 must return to baseline before proceeding.
Ten Incident Quick-Reference Categories
Config Class
Reload no effect – wrong include path or symlink not flipped; nginx -t passes but loads old file. Check: grep process cmdline vs. config server_name.
Two nodes out of sync – Incident 1; cured by "config as code + full push + consistency patrol".
server_name collision – two .conf fight for same domain; first loaded wins, symptom: "change took no effect".
Capacity Class
Connection slots exhausted – worker_connections are not enough; first check if slow upstream is hogging slots, then discuss scaling.
FD exhausted – too many open files; raise both ulimit and worker_rlimit_nofile.
Path Class
502 spike – upstream process restart, keepalive race, wrong port; check error.log for connect() failed vs. reset.
504 cluster – upstream slow or read_timeout too small; use $upstream_response_time to confirm "backend slow" vs. "gateway waited too long".
499 surge – client gave up first; usually P99 jitter, user already retried/closed page; don't just watch 5xx.
Config Trap Class
Cert expiry – zero errors, only client handshake failures; need 30/7/1 day three-tier alerts.
Rate-limit misfire – keyed on NAT egress IP, entire office/ISP blocked; key on uid/token, fallback to IP with relaxed threshold.
Big Promotion Playbooks: T-7 / T-1 / T-0
T-7: capacity at "peak × 1.5" + load test to target; rate-limit thresholds at "normal × 3"; cert validity covers promotion; rehearse one drain + rollback.
T-1: freeze all non-essential changes (including "one-line tweaks"); run patrol script, snapshot baseline; on-call roster & escalation path posted in chat; verify degradation switches (disable cache, disable rate-limit, serve static fallback).
T-0: only three jobs – watch four golden signals, execute playbooks, record timeline.
90% of promotion incidents come from "last-minute config tweaks", not insufficient capacity – freezing changes is the highest-ROI playbook item.
Core Source Code: Four Life-or-Death Logic Paths
3.1 Connection Limit: Why "not enough" Is Logged
// ngx_event_accept()
if (ngx_cycle->connection_n ==
ngx_cycle->free_connections)
{
ngx_log_error(NGX_LOG_ALERT,
log, 0, "conn not enough");
ngx_close_accepted_conn(c);
return;
}Explains three phenomena:
Limit is per-worker – 20,480 × 4 workers = total, but a single worker hitting the ceiling rejects new connections; load tests must watch per-worker free_connections.
Rejection happens after accept – client TCP handshake succeeds then connection is reset; appears as "connection reset" not "connection refused".
Only one ALERT line – doesn't distinguish normal keepalive from slow-upstream hogging; first reaction to this log should be "who is hogging slots", not "raise the number". Raising just moves the wall; the hogging root cause remains.
3.2 Retry Decision: When to Switch Node, When to Give Up
// ngx_http_upstream_next()
if (u->peer.tries == 0
|| (u->conf->next_upstream
& ft_type) != ft_type)
{
ngx_http_upstream_final(r, u);
return; // no more retry
}
ngx_http_upstream_connect(r, u);Two conditions are AND: tries exhausted OR policy disallows this failure type – either stops retry. ft_type is a bitmask; configuring error without timeout means timeouts won't retry – exactly "retry connect errors only, not timeouts" (timeout may mean upstream already processed; retrying a write duplicates orders). Detail: retry re-runs upstream selection – weighted round-robin naturally picks next node, but ip_hash sticks to same node, making retry ineffective – a common cause of "502 never recovers" with session affinity.
3.3 Reload: Why Old Workers Pile Up
// master receives HUP
ngx_spawn_process(cycle, ...);
ngx_signal_worker_processes(cycle,
NGX_SHUTDOWN_SIGNAL);
// old worker: stop accept
// but finish existing connections
if (ngx_exiting && timer->root
== timer->sentinel)
{
ngx_worker_process_exit();
}Explains the reload storm from Part 1: new workers start first to take new connections, old workers get QUIT, stop accepting, but wait for existing connections to finish . Long-lived connections (WebSocket, HTTP/2, 75 s keepalive) pin old workers; frequent reloads stack them layer by layer , each consuming memory and shared memory maps. Three mitigations: ① shorten keepalive_timeout (65 s already conservative); ② use worker_shutdown_timeout to force exit; ③ reduce reload frequency – push high-frequency changes to dynamic gateway (Part 9's value).
3.4 Log Flushing: Why Logs Lag Seconds
// write when buffer full or flush time
if (b->flush || b->pos - b->start
> b->end - b->start)
{
ngx_write_fd(file->fd, b->start,
b->pos - b->start);
b->pos = b->start;
}With access_log buffer=64k flush=2s, logs flush every 64 KB or 2 seconds . Benefit: syscalls drop from per-request to per-second, saving several CPU % at high QPS. Costs: ① 1–2 s tail lag during debugging (don't assume request hasn't arrived); ② last segment lost on crash; ③ disk-full write failures spam error log or block. Hence log rotation + disk watermark alerts are mandatory. Low-traffic services don't need buffering – real-time visibility outweighs tiny CPU savings.
Advanced Topics: Seven Closing Accounts
4.1 SLO & Error Budget: Turn "Stable" Into a Number
Without SLO, stability is a feeling. Define measurable SLO: e.g., "core API 99.9% requests succeed within 300 ms" → 43 min/month downtime budget. Error budget = 1 − SLO ; exhausted → freeze releases, pay down debt; unused → releases too conservative. Gateway implementation: accumulate 5xx rate and P99 monthly; breach auto-freezes pipeline – more effective than any "please be careful" process because stability becomes a quantifiable resource.
4.2 Quarterly Drills: Three Must-Dos
Reload drill – random node reload, verify old workers exit as expected, no business jitter.
SIGKILL drill – kill master, confirm alert fires within 1 min, on-call responds within 5 min.
Drain drill – run §2.2 script to pull one node, verify upstream removal, remaining nodes absorb traffic.
Drill value isn't "prove it works" but "expose assumptions" – the auto-failover you trust, the alert you rely on, many never validated.
4.3 Multi-DC: DNS & Nearest Access
Single-DC HA doesn't survive DC-level failure. Three concerns:
Traffic steering – DNS geo-routing (TTL ≤ 60 s) or Anycast.
Data plane – static assets multi-DC sync or object storage; sessions must be external (Redis or token) – otherwise DC cut = everyone logged out .
Origin pull – every layer must pull independently; no "A DC down but B's gateway still pulls from A".
Litmus test: pull the plug on one DC – does business only degrade, not break? If you can't answer, don't claim multi-active.
4.4 Log Governance: Cost vs. Utility Balance
High QPS → hundreds of GB/day in access.log; log cost often exceeds server cost . Four levers:
Tiering – normal requests sampled/summarized; only 5xx and slow ( $rt > 1s) fully logged.
Field reduction – drop UA, referer; fetch from tracing when needed.
Hot/cold split – local 3 days, object storage 30 days.
Structured JSON – aids querying, but align field names/types or queries drown in casts.
Principle: logs serve debugging, not "peace of mind" ; cost belongs in capacity account.
4.5 Security Baseline: Five Must-Checks
Hide version – server_tokens off.
Upload limits – client_max_body_size + content-type whitelist; block large payloads & executables.
Path traversal – alias must end with slash; otherwise /static../ escapes to parent – classic vuln.
Header injection – don't pass client-controllable Host / X-Forwarded-For via proxy_set_header; use real_ip module from trusted proxies.
Rate limiting – limit_req + limit_conn dual layer against single-IP connection exhaustion.
Security isn't "add a WAF and done" – baseline config is the first gate.
4.6 Team Rules: Three Hard Constraints
Production config changes only via pipeline – SSH read-only; manual edits reverted by patrol.
Every change must declare "impact scope + rollback method + observation metrics" – missing any column blocks release.
On-call roster & escalation path public – anyone at any time knows "who to call now".
Process looks slow but is faster than incidents – the five minutes saved by Incident 1's "one-line edit" cost 40 minutes and a customer complaint.
4.7 When to Change Architecture: Four Signals
These four signals mean single-layer Nginx has hit its ceiling:
Single-node QPS near limit, scaling to dozens of nodes still non-linear → need L4 LB (LVS or cloud).
Routing/gray logic > hundreds of lines of Lua, frequent changes → need programmable gateway (APISIX, etc.).
Service-to-service calls need governance (retry, circuit breaker, mTLS, tracing) → consider Service Mesh.
Changes need sub-second propagation to thousands of nodes → need dedicated control plane.
Conversely, architecting before these signals appear = manufacturing your own failures.
Learning Path: How the 10 Articles Form a System
Layer 1 – Can configure – location/rewrite, reverse proxy/LB, caching/static – covers 80% daily needs; build a full e-commerce gateway in test.
Layer 2 – Can defend – rate-limiting, HTTPS/kernel tuning, logging/troubleshooting – determines if you can locate faults fast; practice "manually create 502/504/499" using Part 2's method.
Layer 3 – Can scale – Keepalived HA, OpenResty dynamic gateway – architecture skills; hands-on dual-node cutover & dynamic upstreams.
Layer 4 – Can operate – this article: capacity, change, fault, cost accounts – wraps first nine parts into process.
Passing grade isn't "read all" but "can solo on-call" – woken at 3 AM, can you tell in 10 minutes whether it's capacity, config, upstream, or network?
20 High-Frequency Interview Questions (with Answers)
The article lists 20 Q&As covering capacity math, FD vs. connection slots, worker_connections exhaustion handling, 502 vs. 504 semantics, 499 danger, dual RT logging, upstream keepalive config, reload worker stacking, config-not-effective causes, canary rationale, timeout setting, retry amplification, 502 triage, cache invalidation, rate-limit misfire, capacity headroom, gateway vs. backend attribution, log volume control, single-node failure transparency, and ops maturity markers. Each answer emphasizes concrete calculation, root-cause hierarchy, and observable signals over vague advice.
One-Page Cheat Sheet (Entire Series)
Condensed reference: Four Accounts (capacity: QPS×RT=concurrency, three ceilings; stability: three timeouts, keepalive stagger, retry budget; observability: four signals + dual RT; change: config-as-code, canary, 60s rollback). Three Ceilings (worker_connections, ulimit -n 2×, memory 8–16 KB/conn). Three Signals (error.log delta, per-node error rate, last change). Five-Step Emergency (stop bleeding → locate → recover → postmortem → improve, single commander). Ten Incident Types (config no-effect, dual-node drift, server_name clash, slot exhaustion, FD exhaustion, 502, 504, 499, cert expiry, rate-limit misfire). Promotion (T-7 load test, T-1 freeze, T-0 watch). Four Architecture-Change Signals (single-node ceiling, Lua sprawl, service governance need, thousand-node sub-second push).
Closing Summary
Nginx in one sentence: simple enough to go live in 30 lines, complex enough to consume a team for a year . Simplicity is syntax; complexity is that every parameter hides an account – connections, timeouts, caches, rate-limits, certs, cutovers, changes, faults. The real divide isn't "can you write location" but "when it breaks, can you name the problem class in 10 minutes". Final takeaways: capacity is calculated, not felt; change has process, not hand-speed; faults have order, not headcount. This system has no finish line – only repeated drills turning it into muscle memory.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
