Troubleshooting Nginx 502, 503, 504 Errors: A Complete Diagnostic Guide

Introduction

Nginx 502, 503, and 504 are the three most common errors in reverse proxy scenarios, yet they are frequently confused. Treating them identically—restarting upstream for 502, restarting Nginx for 503, increasing proxy_read_timeout for 504—only addresses symptoms and can worsen problems (e.g., raising timeout from 60s to 600s turns fast failure into connection pile-up).

Semantics and Origin

502 Bad Gateway : Upstream returned invalid response or closed connection after connect. Origin: Nginx→upstream HTTP layer. Essence: upstream "spoke wrong" or "died mid-stream".

503 Service Unavailable : Nginx rate-limiting rejected, or upstream actively refused. Origin: limit module / upstream app. Essence: "active refusal".

504 Gateway Timeout : Upstream did not respond within timeout. Origin: Nginx waiting for upstream response. Essence: upstream "accepted job but didn't work".

This distinction dictates where to look first: 502 implies connection succeeded (check app logs, crashes, protocol mismatch); 503 means active refusal (check Nginx limit_req/limit_conn or upstream health); 504 means connection established but no response (check slow queries, thread pools, downstream dependencies, GC pauses).

Environment Assumptions

Nginx 1.28.x/1.30.x/1.31.x on Ubuntu 22.04/24.04 or Rocky Linux 9; backends in Java (Spring Boot), Node.js, Go; Nginx as L7 reverse proxy with upstream clusters; custom access_log format including upstream_addr, upstream_status, request_time.

Key Version-Specific Notes

error_log

level must be warn or lower (default) to capture 502/503/504 errors (level error). upstream_status only exists in custom log_format; default combined lacks it. proxy_next_upstream default error timeout retries on connect error/timeout, masking failures via multiple upstream addresses in $upstream_addr. keepalive (pool size) and keepalive_timeout (idle TTL) are separate; both must be set to avoid reusing connections closed by upstream. limit_req_status and limit_conn_status default to 503, making Nginx the primary source of 503.

How Nginx Generates Each Code

Proxy request phases: client → receive request → select upstream → connect → send request → read response header → read response body → client.

Phase [2] select upstream: no live upstreams → 502.

Phase [3] connect: connection refused (ECONNREFUSED) → 502; connect timeout (SYN dropped) → 504.

Phase [4] send request: upstream closes/resets → 502.

Phase [5] read response header: upstream silent/invalid/empty → 504 (timeout) or 502 (protocol error).

Phase [6] read response body: upstream mid-stream close → truncated response.

Connection pool: reused closed keepalive → 502 (retried if configured).

Critical distinction: connect() failed (111: Connection refused) = 502 (upstream port not listening); upstream timed out (110: Connection timed out) = 504 (network unreachable/host hung).

Five-Step Diagnostic Method

Step 0 : Confirm error code distribution and time pattern (e.g., only 502 = app crash; 502+504 mix = crash during processing; daily 3 AM spike = cron/backup).

Step 1 : Extract exact error text from error_log (grep for upstream|connect\(|no live upstreams). Mapping of each error text to meaning and code is provided in the article.

Step 2 : Analyze access_log upstream fields ( ua, us) to pinpoint which upstream returned what; reveals retried-and-masked failures.

Step 3 : Branch by code into dedicated 502/503/504 paths.

Step 4 : Verify fix with three metrics: error count, upstream P99 latency, retry rate (comma in us). Error rate down but retry rate up = problem hidden by retries.

502 Troubleshooting Path

Symptoms & Initial Triage

All requests 502 → upstream all down or config error.

Partial 502 on specific requests → upstream sent too big header or specific endpoint issue.

Intermittent 502, retry succeeds → keepalive reuse or intermittent upstream crash.

Four Mandatory Checks

Upstream listening? ss -lntp; watch for 127.0.0.1-only, IPv6-only with bindv6only=1, Recv-Q buildup.

Connection actually works? nc -zv then curl -sv; TCP ok but HTTP fails = app-layer issue.

Upstream app logs: journalctl -u app or kubectl logs --previous for OOM, panic, RejectedExecution.

Protocol match: HTTPS upstream needs proxy_pass https://; gRPC needs grpc_pass; SNI requires proxy_ssl_server_name on.

Metrics & Monitoring

OSS nginx-prometheus-exporter only exposes stub_status (connections, total requests). For per-code/per-upstream stats, use log exporters (Fluent Bit, Filebeat) or nginx-module-vts (in-memory, real-time). Key insight: 502 absolute count meaningless; compare per-upstream share to detect single sick instance.

Root Cause Categories

Type A: Upstream crash (OOM/panic) → fix memory/bug.

Type B: All upstreams down ( no live upstreams) → check auto-restart (systemd/K8s probes).

Type C: Keepalive connection stale (Nginx keepalive_timeout > upstream idle timeout) → set keepalive_timeout to 1/2–2/3 of upstream idle timeout; add proxy_http_version 1.1 and proxy_set_header Connection "".

Type D: Response header too large ( upstream sent too big header) → increase proxy_buffer_size (e.g., 16k) and audit app for huge cookies/headers.

Type E: Invalid upstream response ( upstream sent invalid header, invalid chunk) → protocol mismatch (h2c vs HTTP/1.1), Content-Length mismatch, middle proxy corruption.

Fix & Verification Example (Type C)

Goal: eliminate off-peak 502, retry rate <1%. Pre-check: current keepalive config, upstream idle timeout (Tomcat connectionTimeout =20s), 502 time distribution (awk hourly). Implement: backup config, adjust upstream block with

keepalive 64; keepalive_timeout 10s; keepalive_requests 1000;

and location with proxy_http_version 1.1; proxy_set_header Connection "";. Test nginx -t, reload. Verify: error log shows reconfiguring, worker PIDs change, 502 count drops, retry rate drops. Rollback: restore backup, reload (allow 60s between reloads).

503 Troubleshooting Path

Identify Source

Parse access_log for ua / us: ua="-" us="-" = Nginx limit; ua="IP" us="503" = upstream app; no 503 in Nginx log but client sees 503 = upstream LB/CDN/WAF.

Nginx-Generated 503 (Rate Limiting)

Default limit_req_status / limit_conn_status = 503. Recommend changing to 429 (Too Many Requests) to avoid triggering LB health checks/failover. Changing status code is a contract change—notify downstream consumers. max_fails default 1 is aggressive; set 3–5 with fail_timeout=10-30s. Open source lacks active health checks; use blackbox_exporter for probing.

Upstream-Generated 503

Thread/connection pool exhaustion: jstack count http-nio threads, check WAITING/BLOCKED; DB pool: SHOW STATUS LIKE 'Threads_connected'.

Startup/shutdown: readiness probe fails; add preStop sleep 10 and terminationGracePeriodSeconds: 60.

Maintenance mode / downstream DB/cache failure: check app logs and dependency connectivity.

Metrics & Fix

Track 503 rate via log exporter or vts. For limit-induced 503: analyze hitting IPs/endpoints; adjust rate / burst or change status to 429. Verify with synthetic load ( curl loop) expecting 429; check error_log for limiting requests, excess: .... Rollback: revert status code, watch for retry storms.

504 Troubleshooting Path

Symptom Classification

Use request_time distribution: if all 504s cluster at fixed value (e.g., 60.001s) = timeout mechanism hit; if scattered = genuine upstream slowness.

Timeout Chain Alignment

Rule: outer timeout > sum of inner timeouts. Layers: client → cloud LB idle timeout → Nginx ( proxy_connect_timeout, proxy_read_timeout, proxy_send_timeout) → app (Tomcat connectionTimeout, Spring read-timeout) → DB (connect/socket timeout) → downstream services. Common misconfig: Nginx 60s, app needs 120s → Nginx cuts off; app 5s, Nginx 60s → app timeout becomes 502/500; client 10s, Nginx 30s → client aborts, Nginx logs client closed connection.

Upstream Slowness Investigation

Java: jstack thread pool saturation, jstat -gcutil GC pause >10% = periodic 504s; DB lock waits ( grep at com.mysql).

Node.js: event loop lag ( nodejs_eventloop_lag p99 >100ms) from sync FS, large JSON, regex backtracking.

Go: DB pool WaitCount / WaitDuration growth = pool exhaustion.

Generic: downstream dependency – tcpdump on upstream to measure real RTT.

Special Scenarios: Long-Lived Connections

SSE/Long-poll:

proxy_buffering off; proxy_cache off; proxy_read_timeout 3600s; chunked_transfer_encoding on;

plus HTTP/1.1 and empty Connection header.

WebSocket:

proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_read_timeout 3600s; proxy_send_timeout 3600s;

– must be separate location from keepalive config.

Metrics

Per-endpoint 504 count and avg latency via awk. Prometheus:

histogram_quantile(0.99, sum by (le, upstream) (rate(nginx_upstream_response_time_bucket[5m])))

. P99 up but P50 stable = long-tail (locks, cache miss); both up = systemic degradation.

Hardening Configurations (All Codes)

Timeouts & Retries

proxy_next_upstream error timeout http_502 http_503; proxy_next_upstream_tries 2; proxy_next_upstream_timeout 10s; proxy_connect_timeout 3s; proxy_send_timeout 30s; proxy_read_timeout 30s; send_timeout 30s;

Caution: http_502 http_503 retries on app-level errors (may be non-idempotent). For POST, never add non_idempotent. proxy_next_upstream_tries default 0 (try all upstreams) amplifies latency during total outage; set 2–3.

Buffers & Body Limits

client_max_body_size 20m; client_body_buffer_size 128k; client_header_buffer_size 4k; large_client_header_buffers 4 16k; proxy_buffer_size 16k; proxy_buffers 8 16k; proxy_busy_buffers_size 32k; proxy_temp_file_write_size 64k; proxy_max_temp_file_size 1024m;

Memory impact: per-connection buffers; calculate worker*connections*buffer_size.

Logging

log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" rt=$request_time uct=$upstream_connect_time uht=$upstream_header_time urt=$upstream_response_time ua="$upstream_addr" us="$upstream_status" host=$host ssl_proto=$ssl_protocol'; access_log /var/log/nginx/access.log main buffer=64k flush=5s; error_log /var/log/nginx/error.log warn;

Buffer reduces IO; flush=5s adds up to 5s log latency. $request_time = total (client upload + Nginx + upstream + client download); $upstream_response_time = upstream only. Large gap = client-side or Nginx processing.

Gzip Pitfalls

Double compression or Content-Length mismatch when upstream already gzipped. Fix:

proxy_set_header Accept-Encoding ""; gzip on; gzip_types application/json;

in location.

Common Pseudo-Fixes (Avoid)

Increase proxy_read_timeout massively → connection pile-up, upstream overload, wasted work after client timeout.

Increase retry count → amplifies traffic on failing upstreams, delays inevitable failure.

Raise max_fails high → failed upstream keeps receiving traffic, slow responses.

Restart Nginx ( systemctl restart) → drops all connections (WebSocket, downloads); use nginx -s reload.

Restart upstream on 502 → destroys evidence (OOM in dmesg/journalctl); capture first, then restart.

Only watch Nginx logs → never see root cause; must correlate with upstream logs.

Risk & Rollback Summary

Operation                Impact                  Risk                                      Rollback
nginx -s reload          All connections (smooth)  Config error: new workers fail, old continue  Restore config, reload
systemctl restart nginx  All connections (hard)    Long connections dropped, state lost        Cannot undo dropped requests
Change proxy_read_timeout Location requests        Too long: pile-up; too short: false timeouts Revert value, reload
Change max_fails         Upstream sensitivity      Too loose: failed node gets traffic         Revert, reload
Change proxy_buffer_size Memory                  OOM if not calculated                     Revert, reload
Change limit status code Client contract         Downstream logic breaks                   Revert to 503
Change worker_connections Per-worker limit         Needs worker_rlimit_nofile & ulimit          Revert, reload

Note: worker_connections limited by worker_rlimit_nofile and system ulimit -n; set worker_rlimit_nofile = worker_connections * 2.

Postmortem Template

Timeline: first occurrence, detection, diagnosis, recovery.

Code distribution: % of 502/503/504, mixed?

Evidence chain: raw error_log lines,

access_log

ua/us

samples, upstream logs same timestamp, metrics (request_time, upstream latency).

Root cause: one sentence "X did Y under condition Z causing …".

Why not detected earlier: monitoring gaps, retry masking, threshold issues.

Improvements: owner, deadline, prioritize automation (alerts, log fields) over "be more careful".

Production Practices

Validate changes in staging with real/replay traffic, then canary.

Record baseline: error rate, retry rate, request_time P50/P99, upstream P99.

Restrict config write access; use GitOps/PR workflow.

Schedule restarts/listen changes in low-traffic windows; reload usually safe but large upstream changes may spike memory.

Rolling reload: one node, observe 10 min, then rest; never batch reload all.

Verify with four metrics vs baseline, not just "page loads".

Summary Judgment Table

Observation                     Primary Check                     Key Command
502 + Connection refused        Upstream listen + app process     ss -lntp, journalctl -u app
502 + Connection reset by peer, Keepalive timeout config        nginx -T, upstream connectionTimeout
502 + too big header            Response header size + buffer     curl -sI, proxy_buffer_size
502 + no live upstreams         max_fails + upstream health     blackbox_exporter, curl /health
503 + ua="-"                   Nginx limit config                limit_req_status, limit_conn_status
503 + us="503"                 Upstream app reject logic         jstack, app logs, pool metrics
504 + rt clinging to timeout    Timeout chain alignment           Compare proxy_read_timeout with upstream timeout
504 + rt scattered              Upstream slow requests            jstack, jstat, downstream tcpdump

Three Actionable Recommendations

Add upstream_addr and upstream_status to access_log – highest ROI, turns guesswork into facts.

Ensure error_log captures error level – one connect() failed (111) beats ten pings.

Document timeout chain (client → LB → Nginx → app → DB) in runbook – 70% of 504 root causes visible there.

Case Studies

Case 1: Off-Peak 3–5 502/hour

Intermittent, self-healing. ua/us showed 10.0.0.11 failing then retry success; error_log had upstream prematurely closed connection. Correlated with K8s pod restarts (47 in 2 weeks) at hourly report job. Root cause: memory limit 512Mi < actual peak 620Mi → OOMKilled. Fix: raise limit to 1Gi, refactor report to streaming. Verification: 72h zero restarts, zero 502. Postmortem: lacked per-instance 502 alert; add 0.5% threshold.

Case 2: Post-Deploy 15% 504 on /api/search

Only one endpoint. request_time all ~60.001s = timeout hit. Upstream metrics: P50 0.08s, P99 59.8s. Root cause: new sync call to recommendation service with 90s default read timeout > Nginx 60s. Fix: set 2s timeout with fallback. Verification: P99 <500ms. Postmortem: missing timeout enforcement in code review; add static check; document timeout chain.

Case 3: 503 Spike, Clean Nginx Logs

Client 503, Nginx access_log shows no 503 increase, request volume -40%. Cloud LB health checks marked both backends abnormal. Root cause: root path changed to 301 redirect; LB health check expected 200. Fix: dedicated /health endpoint returning 200. Postmortem: health check coupled with business endpoint; decouple and review changes for health impact.

Command Cheatsheet

Error time distribution:

awk '$9 ~ /^(502|503|504)$/ {print substr($4,2,14)}' access.log | sort | uniq -c

Upstream + status:

awk '$9 ~ /^(502|503|504)$/ {for(i=1;i<=NF;i++) if($i~/^ua=/) ua=$i; if($i~/^us=/) us=$i; print ua, us}' access.log | sort | uniq -c

Upstream latency percentiles:

grep -oE 'urt=[0-9.]+' access.log | cut -d= -f2 | sort -n | awk '{a[NR]=$1} END {print "p99="a[int(NR*0.99)]}'

Request vs upstream time gap:

awk '{rt=""; urt=""; for(i=1;i<=NF;i++) {if($i~/^rt=/) rt=substr($i,4); if($i~/^urt=/) urt=substr($i,5)} if(rt!="" && urt!="" && urt!="-") printf "%.3f %.3f %.3f
", rt, urt, rt-urt}' access.log | awk '{d=$3; if(d<0) d=-d; if(d>0.5) n++} END {print "gap>0.5s:", n}'

Keepalive active: ss -tn state established '( dport = :8080 )' | wc -l (should be stable near workers*keepalive).

Limit hits:

grep -E 'limiting (requests|connections)' error.log

Upstream Application Responsibilities

Separate /health/live (liveness, no external deps) and /health/ready (readiness, checks DB/cache) – prevents cascade restart on dependency blip.

Log request processing time (e.g., Tomcat %D in access log, Go middleware) to correlate with Nginx $upstream_response_time.

Explicit timeouts for all external calls; total budget = Nginx proxy_read_timeout * 0.8 (e.g., 30s → 24s: DB 40%, cache 10%, downstream 40%, margin 10%). Returning explicit 500 with trace-id beats 504.

Capacity & Concurrency

Three ceilings: worker_processes * worker_connections, worker_rlimit_nofile, system fs.file-max / ulimit -n. Check all three; mismatch logs warning but starts. Each request uses two connections (client + upstream), so effective concurrency ~ worker_connections/2. backlog (kernel accept queue) full → SYN drop → 504; max_conns (Nginx upstream limit) full → queue in Nginx → 504. Distinguish via ss -lnt Recv-Q vs Send-Q. TIME_WAIT buildup can exhaust ports/fds; mitigate with net.ipv4.tcp_tw_reuse=1, ip_local_port_range=1024 65535, somaxconn=4096; never use removed tcp_tw_recycle.

Closing

502/503/504 are result codes, not cause codes. Causes scatter across app, DB, network, kernel. Diagnostic efficiency depends on evidence captured at failure moment, not config mastery. Three habits: enrich access_log, guarantee error_log level, document timeout chain. Then: evidence first, action second; ask "who returned it?" before "why?".