6 Battle-Tested Directions to Diagnose Nginx 502 Errors Fast

Protect the Scene: Don't Restart First

When 502 spikes, restarting Nginx or the app destroys the most valuable evidence: error timestamps, failure types, connection states, and resource pressure. First determine blast radius, then collect the last few minutes of logs. If a large-scale outage is ongoing, run approved emergency playbooks (drain, scale, failover) in parallel; the commands below are for single-node or canary diagnosis, not traffic shifting.

Direction 1: Upstream Process Not Running, Listening on Wrong Path, or Socket Dead

In a Unix-socket stack (systemd → Gunicorn → Unix socket → Nginx), the smoking gun is often in /var/log/nginx/error.log with connect() to unix:… failed plus an errno. Key errnos:

2 (ENOENT) : Nginx config path ≠ actual socket path, or runtime directory not created.

13 (EACCES) : Nginx worker lacks traverse/read-write on socket or any parent directory.

111 (ECONNREFUSED) : Socket file exists but no process listening, or service just exited.

110 (ETIMEDOUT) : More common with TCP upstreams; for local Unix socket, check app hang, queue saturation, or resource starvation first.

Extract decisive upstream lines from error.log, then verify socket and master process — don't trust systemctl is-active alone.

# Code 4: Pull upstream failure lines from error.log
sudo rg -n -i \
  'connect\(\) to |upstream timed out|upstream prematurely closed|recv\(\) failed|no live upstreams' \
  /var/log/nginx/error.log \
  | tail -n 200

Check unit, main PID, and Unix socket simultaneously:

# Code 5: Verify app unit, main PID, and Unix socket coexist
SERVICE_NAME="<app-service-name>"
SOCKET_PATH="<app-run-dir>/gunicorn.sock"

systemctl is-active "$SERVICE_NAME"
systemctl show "$SERVICE_NAME" -p MainPID -p ExecMainStatus -p ActiveEnterTimestamp
ps -fp "$(systemctl show -p MainPID --value "$SERVICE_NAME")"
sudo ss -xlpn | rg -F "$SOCKET_PATH" || true
sudo stat "$SOCKET_PATH"

systemd

active

only means the unit is considered started; for forking/notify/wrapper types, MainPID may not be the real worker. Combine ps, Gunicorn logs, and socket listener to confirm.

Bypass Nginx and hit the socket directly — this is the dividing line: if you can't get a valid HTTP response here, the problem is before Nginx; if this works but Nginx still returns 502, focus on proxy config, permissions, and Nginx worker environment.

# Code 6: Direct health-check via Unix socket
curl --silent --show-error --fail \
  --unix-socket "<app-run-dir>/gunicorn.sock" \
  --max-time 5 \
  -o /dev/null \
  -w 'http_code=%{http_code} total=%{time_total}
' \
  "http://localhost<health-check-path>"

Use localhost only to form the request line; connection goes through --unix-socket. Add -H "Host: <business-domain>" if the app routes by Host. Success only proves this worker can answer this endpoint right now — not that all routes or downstream deps are healthy.

Auditable systemd unit example (replace placeholders):

# Code 7: /etc/systemd/system/<app-service-name>.service key sections
[Unit]
Description=<app-service-name>
After=network.target

[Service]
User=<app-run-user>
Group=<app-run-group>
WorkingDirectory=<app-code-dir>
RuntimeDirectory=<app-run-dir-name>
RuntimeDirectoryMode=0755
ExecStart=<gunicorn-bin> --workers <worker-count> --bind unix:/run/<app-run-dir-name>/gunicorn.sock --access-logfile - --error-logfile - <app-module>
Restart=on-failure
RestartSec=3
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Changing unit or Restart policy is high-risk: it alters start/stop/failure behavior. Back up the current unit, daemon-reload and controlled restart on a canary node first, verify health checks, success rate, worker count, and socket ownership, then roll out.

# Code 8: Safe pre-restart checks after deploy or incident
set -euo pipefail
SERVICE_NAME="<app-service-name>"
systemctl cat "$SERVICE_NAME" > "<evidence-dir>/app-unit-before-action.txt"
systemctl show "$SERVICE_NAME" -p CanReload -p Restart -p TimeoutStopUSec
systemctl status "$SERVICE_NAME" --no-pager
sudo nginx -t
# Only on a drained or healthy-canary node:
# sudo systemctl restart "$SERVICE_NAME"

The restart line is commented because app restart may drop in-flight requests. If the unit defines a proper ExecReload, evaluate whether it truly supports zero-downtime worker replacement — don't assume the command existing means the app implements graceful reload.

Direction 2: Nginx Config Points to Wrong Upstream or Was Overwritten by Deploy

Many 502s happen post-deploy: new app version changes socket to /run/app-v2/gunicorn.sock but Nginx still points to the old path; or multiple include s cause a later location to override the intended one. First find the actually matched server and location, then edit.

# Code 9: Locate proxy_pass, includes, server_name in effective config
sudo nginx -T 2>&1 \
  | rg -n -C 4 'server_name|listen |location |proxy_pass|upstream |include ' \
  > "<evidence-dir>/nginx-routing-extract.txt"

If the same domain appears in multiple server blocks, check listen address, default_server, TLS SNI, and include order. Editing one file that looks relevant may not change the running config at all.

For Unix socket, proxy_pass URI syntax is strict. Example that preserves original URI and passes to socket (do not mix with TCP upstream config in the same location):

# Code 10: Nginx → Gunicorn Unix socket location example
location / {
  proxy_pass http://unix:/run/<app-run-dir-name>/gunicorn.sock:;
  proxy_http_version 1.1;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_connect_timeout 5s;
  proxy_read_timeout 60s;
  proxy_send_timeout 60s;
}

Timeouts must come from business SLA and upstream processing time. Cranking proxy_read_timeout to infinity just holds Nginx worker connections longer — it doesn't fix slow queries or deadlocks. Back up site file before edit; test, then reload a single Nginx and verify with real-Host health check.

# Code 11: Config change backup, syntax check, canary reload, verify
set -euo pipefail
site_file="<nginx-site-file>"
backup_dir="<config-backup-dir>/nginx-$(date +%Y%m%d-%H%M%S)"
sudo mkdir -p "$backup_dir"
sudo cp -a "$site_file" "$backup_dir/"

sudo install -m 0644 "<reviewed-site-file>" "$site_file"
sudo nginx -t
sudo systemctl reload nginx
curl --silent --show-error --fail \
  --resolve "<business-domain>:80:127.0.0.1" \
  -H "Host: <business-domain>" \
  "http://<business-domain><health-check-path>" > /dev/null

nginx -t

before reload is necessary but not sufficient: it checks syntax and partial validity, not socket existence, permissions, or upstream logic. On reload failure, read journalctl -u nginx and error.log for cause.

If error rate rises after change, restore the exact backup file and re-run syntax check. Don't "hand-edit back from memory" — includes, certs, rate limits, security headers are easily missed.

# Code 12: Rollback site file and verify
set -euo pipefail
site_file="<nginx-site-file>"
backup_file="<config-backup-dir>/nginx-<timestamp>/<site-file-basename>"

sudo install -m 0644 "$backup_file" "$site_file"
sudo nginx -t
sudo systemctl reload nginx
sudo nginx -T > "<evidence-dir>/nginx-effective-after-rollback.conf"

<site-file-basename>

is the actual filename saved in the backup dir — don't copy the angle brackets. After rollback, still verify with health check and 5xx rate; if 502 persists, this config wasn't the root cause — avoid thrashing.

Direction 3: Socket or Directory Permissions Block Nginx Worker

Unix socket connect checks more than the socket's mode. The Nginx worker user must have x (traverse) on every parent directory. Common symptom: socket file exists, app curl succeeds, but Nginx error.log reports Permission denied.

# Code 13: Confirm Nginx worker user vs socket ownership/perms
ps -o user=,group=,pid=,cmd= -C nginx
grep -R --line-number --fixed-strings 'user ' /etc/nginx/nginx.conf /etc/nginx/conf.d 2>/dev/null || true
sudo stat -c '%A %a %U:%G %n' "<app-run-dir>" "<app-run-dir>/gunicorn.sock"

Master often runs as root; the worker user is the one that actually touches the socket. Don't assume permissions are fine just because root can curl the socket. If SELinux or AppArmor is enabled, check their denial logs — changing Unix mode won't bypass MAC policy.

# Code 14: Walk each directory level on the socket path
sudo namei -l "<app-run-dir>/gunicorn.sock"
getfacl -p "<app-run-dir>" "<app-run-dir>/gunicorn.sock" 2>/dev/null || true

namei -l

shows per-level perms and owners — direct evidence for "missing x on a directory". getfacl missing doesn't mean no ACL, just tool not installed; don't install packages or tweak ACLs ad-hoc during an incident, follow existing baselines.

If only Nginx worker lacks access, minimal fix is aligning socket group with Nginx worker's effective group and ensuring runtime directory is traversable. Example via app service (don't recursive chmod /run, don't chmod 666 the socket):

# Code 15: Unit snippet to create Gunicorn socket with controlled group
[Service]
User=<app-run-user>
Group=<nginx-accessible-group>
UMask=0007
RuntimeDirectory=<app-run-dir-name>
RuntimeDirectoryMode=0750
ExecStart=<gunicorn-bin> --bind unix:/run/<app-run-dir-name>/gunicorn.sock <app-module>

UMask=0007

affects default perms of all files the process creates — verify logs, upload dirs, temp files don't become unreadable. Safer: check actual socket perms and Nginx access on a staging node first, then promote.

If SELinux is enabled, read denials first — don't disable SELinux. Policies vary by distro and custom modules; command output should be reviewed by security/baseline owner.

# Code 16: Collect SELinux denials only if enabled
getenforce 2>/dev/null || true
sudo ausearch -m AVC -ts recent 2>/dev/null | tail -n 100 || true
sudo journalctl -t setroubleshoot --since "-30 minutes" --no-pager 2>/dev/null || true

If denials confirm SELinux block, apply minimal, auditable file context or policy fix and validate on canary. Setting permissive or disabled widens host attack surface — not a standard 502 fix.

Direction 4: Upstream Alive but Request Times Out, Crashes, or Closes Early

Nginx can connect the socket yet still return 502. Examples: worker crashes before sending response headers → upstream prematurely closed connection; worker OOM-killed, unhandled Python exception, Gunicorn worker timeout, or downstream dependency triggers process exit — all surface similarly.

# Code 17: Correlate Nginx 502, app exceptions, and kernel OOM timeline
sudo journalctl -u "<app-service-name>" --since "-30 minutes" --no-pager \
  | rg -n -i 'error|exception|traceback|worker|timeout|killed|exit' || true

sudo journalctl -k --since "-30 minutes" --no-pager \
  | rg -n -i 'out of memory|oom-killer|killed process' || true

sudo awk '$9 ~ /^502$/ {print}' /var/log/nginx/access.log | tail -n 200

access.log

field order depends on log_format; the awk above assumes default combined with status in column 9. In production, confirm actual log_format; ideally log request_id, upstream_status, upstream_response_time, request_time to stitch a single request end-to-end.

# Code 18: Access log format for correlating 502s
log_format upstream_timing
  '$remote_addr$host "$request" status=$status '
  'request_time=$request_time upstream_addr=$upstream_addr '
  'upstream_status=$upstream_status upstream_time=$upstream_response_time '
  'request_id=$request_id';

access_log /var/log/nginx/access.log upstream_timing;

Changing log_format affects parsing, alerting, and volume. Check log collectors for fixed-field dependencies first; test on one node with reload. Don't flip log format on many nodes mid-incident — adds observability noise.

For a reproducible URI, hit it both via Nginx and direct socket to decide if failure is before or after the proxy. Direct socket probe must use same method, Host, auth headers, and timeout — otherwise comparison is meaningless.

# Code 19: Compare same request via Nginx vs direct socket
request_path="<health-check-path>"

curl --silent --show-error --fail \
  --resolve "<business-domain>:80:127.0.0.1" \
  -H "Host: <business-domain>" \
  -o /dev/null -w 'nginx code=%{http_code} total=%{time_total}
' \
  "http://<business-domain>$request_path"

curl --silent --show-error --fail \
  --unix-socket "<app-run-dir>/gunicorn.sock" \
  -H "Host: <business-domain>" \
  -o /dev/null -w 'socket code=%{http_code} total=%{time_total}
' \
  "http://localhost$request_path"

If direct socket also times out or returns 5xx, Nginx is not to blame — continue with app traces, DB connections, downstream calls, worker count. If direct succeeds but Nginx fails, go back to Direction 2 and 3: effective config, socket path, worker perms.

Gunicorn timeout is a guardrail for workers that don't respond for too long — don't raise it just because "requests are occasionally slow". Find the slow path first, then evaluate request ceiling, async task offload, downstream timeouts, and worker capacity. Below only shows current launch args and process tree.

# Code 20: Inspect Gunicorn worker count, parent/child, actual launch args
SERVICE_NAME="<app-service-name>"
MAIN_PID="$(systemctl show -p MainPID --value "$SERVICE_NAME")"

ps -o pid,ppid,user,etime,stat,cmd --forest -p "$MAIN_PID" --ppid "$MAIN_PID"
tr '\0' ' ' < "/proc/$MAIN_PID/cmdline"
printf '
'

systemctl show "$SERVICE_NAME" -p TasksCurrent -p MemoryCurrent -p LimitNOFILE

Master/worker process relationship depends on launch method; if a wrapper rewrites MainPID, the forest may be incomplete. When worker count is short, investigate why workers exit or can't spawn — don't blindly increase workers when memory is already tight.

Direction 5: Only for TCP Hostname Upstreams — Check DNS, Address, and Connect Path

This section applies only when Nginx config uses proxy_pass http://<upstream-host> or upstream server <upstream-host>:<port>. Unix socket setups have no DNS hop. Common migration issue: app service name, container DNS name, or backend address changed, but Nginx caches old address, or resolver missing so variable proxy_pass can't resolve.

# Code 21: Confirm effective config uses socket, IP, or hostname upstream
sudo nginx -T 2>&1 \
  | rg -n -C 3 'proxy_pass|upstream |resolver |server unix:|server [A-Za-z0-9._-]+:[0-9]+' \
  > "<evidence-dir>/upstream-resolution-extract.txt"

getent ahostsv4 "<upstream-hostname>"
getent ahostsv6 "<upstream-hostname>" || true

getent

queries host's current NSS resolution; it does not guarantee Nginx workers use the same resolver. Nginx resolves static-hostname upstreams at start/reload; variable proxy_pass requires explicit resolver. Judge by effective config — don't rule out DNS just because getent works.

# Code 22: resolver example ONLY for variable-form TCP upstream
resolver <dns-server-ip> valid=30s ipv6=off;
resolver_timeout 5s;

set $backend "http://<upstream-hostname>:<upstream-port>";
location / {
  proxy_pass $backend;
  proxy_connect_timeout 3s;
  proxy_read_timeout 60s;
}

<dns-server-ip>

must be a DNS server reachable by Nginx workers and managed by ops — don't pick random public DNS. Disable IPv6 only when upstream has no usable IPv6 and evidence shows AAAA resolution or IPv6 path issues. Changing resolver affects dynamic upstream resolution in that server — canary first.

Even with DNS healthy, TCP upstream may not listen, be firewalled, or exhaust connection pool. Probe from the Nginx node; confirm network policy allows it, don't hit write-management ports by mistake.

# Code 23: Verify TCP upstream address, port, and HTTP health
upstream_host="<upstream-hostname>"
upstream_port="<upstream-port>"

getent ahostsv4 "$upstream_host"
nc -vz -w 3 "$upstream_host" "$upstream_port"
curl --silent --show-error --fail \
  --connect-timeout 3 \
  --max-time 5 \
  "http://$upstream_host:$upstream_port<health-check-path>" > /dev/null

nc

and curl success only proves this node can reach one resolved IP right now. Multiple A records, L4 LB, IPv6, network partitions, and app Host routing still need individual verification. If config uses a service-discovery name, investigate from the service-discovery system and runtime network — don't treat a temporary /etc/hosts edit as a proper fix.

Direction 6: Host Resource Exhaustion and Cascading Failures

When CPU saturates, memory nears cgroup limit, FDs exhausted, disk full, kernel conntrack full, or per-host connection count spikes, the app may not hard-crash but can't accept, fork, or respond in time. These 502s often show peaks and brief recovery after restart; restart merely frees resources — it doesn't explain why they were exhausted.

# Code 24: One-shot check of CPU, memory, disk, FD, connections, OOM evidence
SERVICE_NAME="<app-service-name>"
MAIN_PID="$(systemctl show -p MainPID --value "$SERVICE_NAME")"

uptime
free -h
df -hT
df -ih
grep -E 'Max open files' "/proc/$MAIN_PID/limits"
ls "/proc/$MAIN_PID/fd" | wc -l
ss -s
cat /proc/sys/fs/file-nr
sudo journalctl -k --since "-1 hour" --no-pager \
  | rg -i 'oom|out of memory|killed process|nf_conntrack.*full' || true

Resource conclusions need trend support. E.g., df -h shows a filesystem full and app logs simultaneously show "cannot write log/temp file" → disk is root cause; high CPU alone doesn't distinguish normal high throughput from hot spots, lock contention, or softirq storms.

If systemd memory limits are used, check unit's current limit vs actual usage. Changing memory limit alters OOM risk — validate on canary first, keep original unit. Don't disable limits to hide leaks.

# Code 25: Check app cgroup memory, tasks, OOM fields
SERVICE_NAME="<app-service-name>"
systemctl show "$SERVICE_NAME" \
  -p MemoryCurrent -p MemoryMax -p MemoryHigh \
  -p TasksCurrent -p TasksMax \
  -p LimitNOFILE

MAIN_PID="$(systemctl show -p MainPID --value "$SERVICE_NAME")"
cat "/proc/$MAIN_PID/cgroup"

MemoryCurrent

showing unavailable or empty may stem from cgroup version, permissions, or unit type — don't interpret empty as unlimited. Cross-check with container runtime or cgroup path files like memory.events, memory.current.

One Executable 502 Triage Sequence

In an incident, the goal is shortening the error branch, not collecting more irrelevant commands. The script below does read-only forensics and local probes — no reloads or restarts. Suitable for a single suspect node to produce the first evidence pack. Ensure evidence directory perms and disk space beforehand.

#!/usr/bin/env bash
# Code 26: Nginx 502 first-round read-only evidence script
set -euo pipefail

SERVICE_NAME="<app-service-name>"
SOCKET_PATH="<app-run-dir>/gunicorn.sock"
evidence_dir="<evidence-dir>/nginx-502-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$evidence_dir"

systemctl status nginx --no-pager > "$evidence_dir/nginx-status.txt" || true
systemctl status "$SERVICE_NAME" --no-pager > "$evidence_dir/app-status.txt" || true
sudo tail -n 500 /var/log/nginx/error.log > "$evidence_dir/nginx-error.log" || true
sudo journalctl -u "$SERVICE_NAME" --since "-30 minutes" --no-pager > "$evidence_dir/app-journal.txt" || true
sudo ss -xlpn > "$evidence_dir/unix-sockets.txt" || true
sudo stat "$SOCKET_PATH" > "$evidence_dir/socket-stat.txt" || true
free -h > "$evidence_dir/memory.txt"
df -hT > "$evidence_dir/disk.txt"
printf '%s
' "$evidence_dir"

Commands that fail are tolerated with || true so missing socket or stopped service still leaves other evidence — this is not ignoring errors. During analysis, check each command's exit status and combine "socket missing", "service inactive", "error.log errno", and "direct socket result" into one conclusion.

Post-triage validation should be layered: first confirm Nginx-to-socket health check passes, then verify canary traffic shows no upstream failures, finally confirm error rate, P95/P99, app worker count, and system resources return to expected ranges. If config must roll back, use Direction 2's exact backup file and nginx -t first. If app must restart, drain first, confirm healthy replicas, roll one node at a time, watch traffic backfill. Only this way does a 502 leave a reproducible root cause — not a "restarted and it worked" fluke.