Nginx Logging Deep Dive: $request_time vs $upstream_response_time, Log Rotation Pitfalls, and Production Troubleshooting
This article analyzes three real Nginx incidents to teach production log design, log rotation via USR1 signal, and a three-step time-difference method to pinpoint latency across client, Nginx, and upstream layers using $request_time, $upstream_connect_time, and $upstream_response_time.
Three Incidents, Three Lessons
The article opens with three production incidents that frame the core lessons:
Incident 1: rm access.log deleted the file but disk space wasn't released because Nginx workers still held the file descriptor. The fix: use logrotate with postrotate kill -USR1 to trigger reopen.
Incident 2: An order interface P99 jumped from 200ms to 2s. Default log format lacked $upstream_response_time, so nobody could tell if Nginx or upstream was slow. Adding the variable revealed request_time=2s, upstream_response_time=1.98s — upstream was the bottleneck.
Incident 3: A spike in 499 status codes was misclassified as an attack, leading to 2000 IPs blocked. Root cause: gateway proxy_read_timeout=60s while mobile app timeout was 30s. Clients gave up first; upstream queries kept running. 499 means "client left early," not attack.
1. Technical Analysis: Four Pillars of the Log System
1.1 Two Log Streams: Requests Speak, Processes Speak
access_log — "request speaks": one line per HTTP request at completion, answering what happened, result, latency.
error_log — "process speaks": internal events (connection refused, upstream timeout, no live upstreams).
Common mistakes: only watching access_log, or setting error_log to crit to reduce noise — blinds you to early warnings.
1.2 error_log Levels and Granular Control
Levels: debug, info, notice, warn, error, crit, alert, emerg. Rule: "≥ configured level gets logged." Production recommendation: warn or error. info is too verbose (logs every upstream attempt); crit misses precursors. debug requires compile-time --with-debug. error_log can be set per context (main, http, server, location) — enable info on a single location for surgical debugging.
1.3 Troubleshooting Variable Checklist: What You Log Determines What You Can Debug
A production log_format must include four variable groups:
Identity: $remote_addr, $http_x_request_id (generate $request_id if missing, propagate upstream) — the key to join Nginx, app, and DB logs.
Result: $status, $body_bytes_sent, $request (method + URI).
Time: $time_local, $request_time.
Upstream: $upstream_addr, $upstream_status, $upstream_connect_time, $upstream_response_time. Default combined format lacks upstream group entirely — root cause of Incident 2.
Special behavior: $upstream_addr becomes comma-separated on retry (e.g., "10.0.0.1:80, 10.0.0.2:80") with matching $upstream_status ( "504, 200") — a complete retry trail in one line. $request_time includes slow-client transfer time; $upstream_response_time isolates upstream processing. Their difference ≈ Nginx overhead + network transport.
1.4 access_log Overhead and Buffer Switches
Each log line triggers a write syscall at request end. At 100k QPS that's 100k writes/sec. Page cache absorbs most, but disk jitter or container I/O throttling turns logging into P99 tail latency. Buffer parameters mitigate: buffer=64k flush=5s batches writes, reducing syscalls by two orders of magnitude. Trade-off: crash loses last few seconds; debugging requires waiting for flush — temporarily set flush=1s during incidents.
Conditional logging via map drops health checks, static assets, CDN probes. open_log_file_cache caches file descriptors to avoid per-line open/close, but hides newly created log files within its valid window — dangerous if external tools rotate by minute. Default off is safer; enable only for fixed, high-volume log files.
1.5 error_log Reading: Every Line Carries Coordinates
Each line has timestamp, level, source (module:function), and client/upstream/request context. Read backwards — faults accumulate, earliest anomalies appear first. Three classic entries:
connect() failed (111: Connection refused) while connecting to upstream— upstream port not listening.
upstream timed out (110: Connection timed out) while reading response header— connected but response header timeout ( proxy_read_timeout), often GC pause or slow SQL. no live upstreams while connecting to upstream — all peers marked down via max_fails.
1.6 Five Status Codes as Diagnostic Semantics
502 : upstream returned invalid response or connection failed (crash, incomplete response, TLS handshake failure).
504 : upstream timeout — proxy_connect_timeout (connect) or proxy_read_timeout (read). Distinguish via error_log phase.
499 : client closed before response sent (client timeout or page close). Nginx usually lets upstream finish.
408 : client too slow sending request headers (weak network).
500 : Nginx internal or config error.
Status distribution is a dashboard: 502 clustered on one node = node failure; 504 globally up = upstream slowdown; 499 spike = slow endpoints or client timeout shortened.
1.7 Time-Difference Localization: Three Steps to the Layer
Compare $upstream_connect_time — large = connection setup issue (DNS, cross-DC, pool exhaustion).
Compare $upstream_response_time — large = upstream processing slow; take x_request_id to app logs and slow query.
Both small but $request_time large = Nginx or response transfer: large body, slow client, gzip CPU, or unbuffered log write jitter.
Prerequisite: all variables present in log format — localization ceiling set the day you configure logging.
2. Core Workflow: From Config to Real Investigation
2.1 Production Config #1: Log Format & Rotation
http {
log_format main
'$remote_addr - [$time_local]'
'"$request" $status $body_bytes_sent'
'rt=$request_time'
'urt=$upstream_response_time'
'uct=$upstream_connect_time'
'ua=$upstream_addr us=$upstream_status'
'rid=$http_x_request_id';
map $status $loggable {
~^[23] 0; # 2xx/3xx sample 10%
default 1; # rest full log
}
access_log /var/log/nginx/access.log main buffer=64k flush=5s;
error_log /var/log/nginx/error.log warn;
}
# logrotate: /etc/logrotate.d/nginx
/var/log/nginx/*.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
postrotate
kill -USR1 $(cat /run/nginx.pid)
endscript
}Key points mapped to incidents:
Time + four upstream variables complete → three-step diff works (Incident 2). map conditional logging: 2xx sampled 10% (hash-based for trace continuity), errors fully logged — volume down 10x without losing debuggability. buffer+flush eliminates high-concurrency write tail. logrotate uses USR1 not copytruncate — latter has a copy/clear race window; former atomically swaps fd (Incident 1). rotate 14 keeps two weeks; delaycompress leaves latest uncompressed for immediate grep.
2.2 USR1 Reopen Full Timeline
T1: logrotate mv access.log access.log.1 — same-disk rename, inode unchanged, Nginx fd still points to it, writes continue to old file.
T2: logrotate sends USR1 to master.
T3: Master reopens all log files per config, gets new inodes.
T4: Master broadcasts NGX_CMD_REOPEN via channel to all workers.
T5: Workers swap fd tables; subsequent writes go to new file; old file receives no new writes, safe to compress/delete.
Missing T2 reverts to Incident 1: stale fd holds deleted inode, disk fills. Emergency: kill -USR1 or nginx -s reload (implicit reopen) releases space instantly — no process restart needed.
2.3 Production Config #2: Troubleshooting Toolbox
http {
// 1. Request ID: generate at entry, propagate upstream, chain logs
map $http_x_request_id $rid {
default $http_x_request_id;
'' $request_id;
}
server {
location /api/ {
proxy_set_header X-Request-Id $rid;
proxy_pass http://app_pool;
// 2. Debug headers on error responses
proxy_intercept_errors off;
add_header X-Debug-Upstream $upstream_addr always;
add_header X-Debug-Status $upstream_status always;
// 3. Per-location fine-grained error_log
error_log /var/log/nginx/api.log info;
}
}
}Three components:
request_id chaining — client-provided or generated, passed upstream; app logs print same ID. Join Nginx + app + DB logs with one key — cheapest distributed tracing infrastructure.
Debug headers — on 502/504, response carries X-Debug-Upstream (which node) and X-Debug-Status (what it returned). always ensures headers on error responses. Remove after incident.
Location-level error_log — only critical endpoints get info, avoiding global noise. All three are "harmless normally, lifesaving when needed"; keep 1 and 2 permanently.
2.4 Slow Request Investigation Timeline
T1: Business reports slowness; obtain request_id (client instrumentation or user replay).
T2: grep access.log for the line; extract rt, urt, uct, ua.
T3: Three-step diff — uct large → network/pool; urt large → take rid to app logs; both small → Nginx transfer (large body, slow client, gzip, log write).
T4: If urt large, scan nearby lines for same endpoint — isolate "one-off" (large object, bad param) vs "pattern" (cron overlap, hourly job).
T5: Cross-check error_log for concurrent upstream timed out entries.
T6: Post-fix verification — same endpoint urt P95 recovered?
Entire flow: no restart, no config change, no deploy — pure log reading. That's the ROI of log format investment.
2.5 Archiving & Central Collection Pitfalls
Buffer vs collector interaction: Filebeat tails file; buffered writes are batch; agent's registry offset may mismatch after rotate+compress. Mitigation: delaycompress (latest stays plaintext) + give agent read window before rotate.
Container double-write: Logging to both file and stdout causes duplicate lines in search. Standardize: either all stdout (k8s sidecar collects) or all file (volume mount), and disable the other at image level.
3. Core Source Code: Four Critical Logic Paths
3.1 Log Write Path: Failure Never Blocks Response
// ngx_http_log_module.c
static ngx_int_t ngx_http_log_handler(r) {
for each log conf in r->http_log_logs {
len = ngx_http_log_script_format(r); // render vars → buffer
if (conf->buffer) {
append(buf, line);
if (full or expired) flush(buf);
} else {
ngx_write_fd(fd, line);
}
// write failure: only error_log alert
// NEVER affects response
}
}Two design decisions:
Logging runs in LOG phase — last of 11 pipeline phases, after response fully sent. Thus $upstream_* variables hold final values (including all retries).
Write failure silently degrades — disk full only logs alert to error_log; user response unaffected. Consequence: log disk saturation goes unnoticed by business; must monitor disk usage independently.
3.2 Reopen Signal Handling: Atomic FD Swap
// ngx_process_cycle.c
static void ngx_signal_handler(sig) {
if (sig == SIGUSR1) ngx_reopen = 1;
}
// main loop next iteration
if (ngx_reopen) {
ngx_reopen = 0;
ngx_reopen_files(cycle, -1); // reopen all log files
ngx_channel = NGX_CMD_REOPEN; // broadcast to workers
}Reopen implements "fd swap" as broadcast protocol: master opens new files first, then signals workers; each worker closes old fd, adopts inherited new fd. Writes never pause — last few lines before command go to old inode, new writes immediately use new file; at most one line lost, zero requests lost. This is why logrotate standard is "mv then USR1": mv preserves inode (safe), USR1 atomically swaps. reload (SIGHUP) implicitly reopens — hence emergency reload frees deleted space.
3.3 499 Generation: Client Leaves, Server Cleans Up
// ngx_http_request.c
static void ngx_http_request_handler(ev) {
if (ev->closed) { // EPOLLRDHUP / read returns 0
r->closed = 1;
r->status = NGX_HTTP_CLIENT_CLOSED; // 499
// upstream request continues by default
// (unless ignore_client_abort on)
}
}499 is Nginx-custom (not in HTTP spec): "client closed before response sent." Two source-level takeaways:
It's a marker, not a circuit breaker — default config lets upstream run to completion (consumes upstream connection). Hence "massive 499" doesn't relieve upstream pressure — the slow query keeps running (Incident 3 mechanism). proxy_ignore_client_abort on can abort upstream on client disconnect, but changes retry and billing semantics (payment upstreams must never abort mid-transaction). Default is safer; real fix is lowering urt.
3.4 upstream_addr Multi-Value: Retry Trail Recorder
$upstream_addrrecords multiple values because upstream module maintains a per-request attempt array: each next_upstream trigger appends "address, status, latency". Log rendering outputs sequentially. A line like ua=10.0.0.1:80, 10.0.0.1:80, 10.0.0.2:80 us=504, 504, 200 tells the full story: same node timed out twice, third node succeeded. When investigating "occasional slow requests," any multi-value ua goes to retry analysis first — retry adds at least one full attempt latency. Many "interface occasionally 2x slow" tickets root-cause to a jittery node triggering next_upstream. Conversely, frequent multi-value ua is direct evidence to tune max_fails / fail_timeout or evict bad nodes.
4. Advanced Topics: Six Accounts to Settle Upfront
4.1 Massive 499 Classification Triage
499 spike → classify before acting:
Large urt — upstream slow, client timeout shorter → fix upstream.
Small urt , concentrated on specific endpoints — user behavior (bounce, preload cancel) → normal, whitelist from error-rate metrics.
Small urt , random, highly distributed source IPs — crawlers/scanners actively closing → rate limit.
Dashboards must separate 499 from 5xx; mixing inflates capacity alerts (false high) and misdirects debugging (think service crashed). Incident 3's lesson: no classification → blanket "attack" verdict → real slow endpoints keep hurting users.
4.2 Log Volume Governance: Sampling & Split Files
High-traffic node math: 100k QPS × 500B/line ≈ 50GB/hour → TB/day. Three levers:
Status-code tiering: 2xx sample (hash request_id for 10%, ensuring same request either fully logged or fully absent across hops); 4xx/5xx full log.
Request-type split files: Health checks, static assets → separate access_log or discard; business endpoints exclusive → no noise during search.
Field trimming: $request query strings with signatures are long and sensitive; use map to truncate. Sampling must hash by ID, not random : same request either has full trace or none, else trace breaks at Nginx hop.
4.3 Data Masking: Logs Are the Largest Sensitive Data Leak Surface
Access logs default to full URI + partial headers; URIs often contain phone, ID card, token. Compliance-wise this exceeds databases (logs replicated to multiple collectors, retained 14-30 days, wider permissions). Low-cost fix: log_format omits raw $request; use map to produce safe version — keep path, whitelist query params; exclude Authorization / Cookie headers. Better to lose some debug detail than make logs a compliance violation. Mask at write time, not post-collection — every on-disk copy is a liability.
4.4 Disk Full Emergency & Prevention
Emergency three steps: lsof | grep deleted find deleted-but-held files (Incident 1); kill -USR1 releases.
If disk genuinely full (not deleted), temporarily point access_log to /dev/null and reload — not comment out (comment falls back to default path, may write elsewhere).
After expansion, restore config and review capacity model.
Prevention three measures:
Log disk on independent partition (isolated from system/data disks).
Usage >80% alert. rotate params reconciled with traffic — estimate daily growth from current QPS; 14-day retention must be calculable, not guessed.
4.5 Packet Capture & Mirroring: Where Logs Can't See
Logs capture metadata, not body. When issue is in request/response body:
tcpdump upstream interaction (port-filtered to upstream ports) — see TCP retransmits vs RST, distinguish network vs app.
mirror module — async copy traffic to debug backend, full request body preserved for intermittent param bugs (fire-and-forget, no main-path impact; debug backend must be rate-limited).
strace attach worker — syscall view: log write stalls, abnormal connect behavior — issues invisible at log layer. All three are "low frequency, high impact"; use and remove, not permanent.
4.6 Six High-Frequency Pitfalls
rm then assume space freed — fd open, must USR1 /reload.
Buffer debugging trap — buffer on, log tail lags; during debug set flush=1s or disable buffer.
debug ineffective — missing --with-debug at compile; config does nothing.
$remote_addr is CDN edge IP — without real_ip module, all sources appear as edge nodes; blocking and profiling all distorted (see Episode 03 XFF trust chain).
access_log inside if — if is directive block, not toggle; conditional logging uses map variable (empty string skips).
error_log inside if — not allowed; nginx -t catches it immediately.
5. Learning Path: From Reading Logs to Building Troubleshooting Systems
Upgrade your machine's logs to "production grade" — rewrite log_format per 2.1, create slow requests (upstream sleep), run three-step diff, experience "logs speak" end-to-end.
Read ngx_http_log_module write path and ngx_reopen_files signal handling; spend 30 minutes on official variable reference — $upstream_*, $request_* families are your debugging vocabulary; vocabulary size determines expressive power.
Forward to HA (next: Keepalived), backward to centralized collection (ELK/Loki) — upgrade "single-machine logs" to "searchable troubleshooting system."
6. High-Frequency Interview Questions (with Answers)
The article includes 20 Q&A covering: deleted log space not freed, logrotate USR1 flow, $request_time vs $upstream_response_time, 502 vs 504, 499 meaning and handling, access_log buffer trade-offs, error_log level selection and debug compile requirement, $upstream_addr multi-value meaning, log volume control, sensitive data handling, if pitfalls, open_log_file_cache caveats, "no live upstreams" diagnosis, request_id chaining, proxy_ignore_client_abort pros/cons, Filebeat loss prevention, disk full emergency, $upstream_connect_time usage, mirror module use cases, production error_log config.
7. One-Page Cheat Sheet
Two streams: access=request, error=process — decide which to read first.
Format four groups: identity (addr+rid), result (status+bytes), time (request_time), upstream (addr/status/connect/response) — missing variables = blind layers.
Three-step diff: connect_time large → connection; response_time large → upstream; both small → transport.
Rotation iron law: mv then USR1; rm doesn't release fd.
Status code semantics: 502=upstream bad, 504=upstream slow, 499=client left early (upstream still running), 408=client sent slow.
Multi-value ua: retry trail; intermittent slowness → check it first.
Four standing tools: rid chaining, debug headers, location-level error_log, log disk alert.
8. Summary & Next Episode Preview
Log system value isn't "how much recorded" but "can we, within three minutes, cut latency to layer and pinpoint error to line." Recap:
FD lifecycle decides space release — rotation via reopen is discipline, not trick.
Format is troubleshooting ceiling — four variable groups enable three-step diff.
Status codes are semantics, not numbers — 499 reads as "upstream exceeded client patience."
Next: HA — Keepalived + VIP active-passive: how split-brain happens, VRRP priority/preemption trade-offs, writing Nginx health-check scripts that don't false-failover, and cloud-era alternatives to primary-backup.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
