6 Essential Steps to Diagnose Nginx 502 Errors

502 Bad Gateway means Nginx received a client request but did not obtain a usable HTTP response from the upstream. It does not guarantee that the application is down or that restarting Nginx will fix the problem. The article uses a concrete production stack: systemd manages Gunicorn, which serves a Python WSGI app via a Unix socket, and Nginx reverse‑proxies that socket.

Preserve the scene – do not restart first

When a 502 surge appears, restarting Nginx or the application discards valuable evidence such as error timestamps, failure types, connection states and resource pressure. First determine the impact scope and collect logs from the recent minutes.

# Confirm Nginx version, main process and config path
nginx -v
nginx -V 2>&1
systemctl status nginx --no-pager
sudo nginx -T > "<evidence_dir>/nginx-effective.conf"

# Collect logs from the 502 time window
time_window="15 minutes ago"
sudo journalctl -u nginx --since "$time_window" --no-pager > "<evidence_dir>/nginx-journal.txt"
sudo journalctl -u "<app_service_name>" --since "$time_window" --no-pager > "<evidence_dir>/app-journal.txt"
sudo tail -n 500 /var/log/nginx/error.log > "<evidence_dir>/nginx-error-tail.txt"

Direction 1 – upstream process not running, wrong listen position, or socket invalid

In a Unix‑socket setup the most common evidence is an error.log line like connect() to unix:… failed with an errno. The article lists the meanings of the most frequent error numbers:

2 – No such file or directory : Nginx configuration points to a path that does not exist or the socket was not created after the service started.

13 – Permission denied : the Nginx worker lacks read/write or traverse permission on the socket or its parent directories.

111 – Connection refused : the socket file exists but no process is listening, or the service has just exited.

110 – Connection timed out : more common for TCP upstreams; for a local socket it usually indicates a hung worker or resource starvation.

# Extract decisive upstream error lines
sudo rg -n -i 'connect\(\) to |upstream timed out|upstream prematurely closed|recv\(\) failed|no live upstreams' /var/log/nginx/error.log | tail -n 200

# Verify socket existence and listening process
SERVICE_NAME="<app_service_name>"
SOCKET_PATH="<app_dir>/gunicorn.sock"
systemctl is-active "$SERVICE_NAME"
systemctl show "$SERVICE_NAME" -p MainPID -p ExecMainStatus -p ActiveEnterTimestamp
ps -fp "$(systemctl show -p MainPID --value "$SERVICE_NAME")"
sudo ss -xlpn | rg -F "$SOCKET_PATH" || true
sudo stat "$SOCKET_PATH"

Note that systemctl is-active only reports the unit state; it does not guarantee that the socket is usable. Use ss -xlpn and stat to confirm the socket and its permissions.

Direction 2 – Nginx configuration points to the wrong upstream or is overwritten by a release

Many 502 incidents occur after a deployment when the new version changes the socket path or upstream address but Nginx still references the old configuration. Locate the effective server and location blocks, then verify proxy_pass, include and listen directives.

# Extract proxy_pass, include and server_name from the effective config
sudo nginx -T 2>&1 | rg -n -C 4 'server_name|listen|location|proxy_pass|upstream|include' > "<evidence_dir>/nginx-routing-extract.txt"

Example of a correct location block for a Unix socket (replace placeholders with the actual paths):

location / {
    proxy_pass http://unix:/run/<app_runtime_dir>/gunicorn.sock:;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_connect_timeout 5s;
    proxy_read_timeout 60s;
    proxy_send_timeout 60s;
}

Before applying changes, back up the site file, test the syntax with nginx -t, reload a single node, and verify health checks.

Direction 3 – socket or directory permissions block Nginx workers

Even if the socket file exists, Nginx workers need execute (traverse) permission on every parent directory. Inspect the worker user, group, and socket ownership, then use namei -l and getfacl to pinpoint missing directory permissions.

# Identify Nginx worker user and group
ps -o user=,group=,pid=,cmd= -C nginx
grep -R --line-number --fixed-strings 'user ' /etc/nginx/nginx.conf /etc/nginx/conf.d 2>/dev/null || true

# Show permissions of the socket directory and file
sudo stat -c '%A %a %U:%G %n' "<app_dir>" "<app_dir>/gunicorn.sock"
namei -l "<app_dir>/gunicorn.sock"
getfacl -p "<app_dir>" "<app_dir>/gunicorn.sock" 2>/dev/null || true

If SELinux or AppArmor is enabled, collect AVC denial logs with ausearch and journalctl -t setroubleshoot instead of disabling the security module.

Direction 4 – upstream application alive but times out, crashes, or closes early

Nginx may still return 502 if the upstream worker crashes before sending headers, is killed by the OOM killer, or hits its Gunicorn timeout. Correlate Nginx 502 entries with application logs, kernel OOM messages, and access‑log fields that include $upstream_status and timing information.

# Correlate 502 entries with application errors and OOM events
sudo journalctl -u "<app_service_name>" --since "-30 minutes" --no-pager |
    rg -i 'error|exception|traceback|worker|timeout|killed|exit' || true
sudo journalctl -k --since "-30 minutes" --no-pager |
    rg -i 'out of memory|oom-killer|killed process' || true
# Extract recent 502 responses from access.log (default combined format)
sudo awk '$9 ~ /^502$/ {print}' /var/log/nginx/access.log | tail -n 200

Compare a request that goes through Nginx with a direct socket request to isolate whether the failure lies in Nginx or the upstream.

# Nginx‑proxied request (preserve Host header)
curl --silent --show-error --fail \
    --resolve "<business_domain>:80:127.0.0.1" \
    -H "Host: <business_domain>" \
    -o /dev/null \
    -D "<evidence_dir>/response-headers.txt" \
    -w 'http_code=%{http_code} connect=%{time_connect} starttransfer=%{time_starttransfer} total=%{time_total}
' \
    "http://<business_domain><health_path>"

# Direct Unix‑socket request (bypass Nginx)
curl --silent --show-error --fail \
    --unix-socket "<app_dir>/gunicorn.sock" \
    --max-time 5 \
    -o /dev/null \
    -w 'http_code=%{http_code} total=%{time_total}
' \
    "http://localhost<health_path>"

Direction 5 – TCP hostname upstream – DNS, address and connection path

If the upstream is a TCP hostname, Nginx resolves hostnames at start‑up (or via resolver for variables). Verify the effective configuration, query the system resolver, and test connectivity.

# Determine whether the config uses a socket, IP, or hostname
sudo nginx -T 2>&1 | rg -n -C 3 'proxy_pass|upstream|resolver|server unix:|server [A-Za-z0-9._-]+:[0-9]+' > "<evidence_dir>/upstream-resolution-extract.txt"

# Resolve the hostname using the host's NSS resolver
getent ahostsv4 "<upstream_hostname>"
getent ahostsv6 "<upstream_hostname>" || true

# Test TCP connectivity and HTTP health endpoint
nc -vz -w 3 "<upstream_host>" "<upstream_port>"
curl --silent --show-error --fail --connect-timeout 3 --max-time 5 "http://<upstream_host>:<upstream_port><health_path>" > /dev/null

Direction 6 – host resource exhaustion and cascading failures

CPU saturation, memory limits, file‑descriptor exhaustion, full disks, or kernel connection‑table limits can cause 502 even if the application process is still running. The article provides a one‑shot script that captures CPU load, memory, disk usage, FD usage, socket statistics and recent OOM or conntrack messages.

# Collect resource usage and OOM evidence
SERVICE_NAME="<app_service_name>"
MAIN_PID="$(systemctl show -p MainPID --value "$SERVICE_NAME")"

uptime
free -h
df -hT
df -ih
grep -E 'Max open files' "/proc/$MAIN_PID/limits"
ls "/proc/$MAIN_PID/fd" | wc -l
ss -s
cat /proc/sys/fs/file-nr
sudo journalctl -k --since "-1 hour" --no-pager |
    rg -i 'oom|out of memory|killed process|nf_conntrack.*full' || true

Check cgroup memory limits, task counts and ensure that observed trends (e.g., a full filesystem) align with log evidence before concluding a resource‑related root cause.

One‑executable 502 handling script

The following Bash script gathers all the evidence above without performing any reload or restart. It is suitable for a single problematic node.

#!/usr/bin/env bash
set -euo pipefail

SERVICE_NAME="<app_service_name>"
SOCKET_PATH="<app_dir>/gunicorn.sock"
EVIDENCE_DIR="<evidence_dir>/nginx-502-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$EVIDENCE_DIR"

systemctl status nginx --no-pager > "$EVIDENCE_DIR/nginx-status.txt" || true
systemctl status "$SERVICE_NAME" --no-pager > "$EVIDENCE_DIR/app-status.txt" || true
sudo tail -n 500 /var/log/nginx/error.log > "$EVIDENCE_DIR/nginx-error.log" || true
sudo journalctl -u "$SERVICE_NAME" --since "-30 minutes" --no-pager > "$EVIDENCE_DIR/app-journal.txt" || true
sudo ss -xlpn > "$EVIDENCE_DIR/unix-sockets.txt" || true
sudo stat "$SOCKET_PATH" > "$EVIDENCE_DIR/socket-stat.txt" || true
free -h > "$EVIDENCE_DIR/memory.txt"
df -hT > "$EVIDENCE_DIR/disk.txt"
printf '%s
' "$EVIDENCE_DIR"

During analysis, match error‑log timestamps with access‑log 502 ratios, verify socket existence and permissions, confirm DNS resolution matches Nginx’s resolver configuration, and ensure that resource metrics do not indicate exhaustion. Only after the root cause is identified should configuration changes be applied, backed up, syntax‑checked with nginx -t, and rolled out gradually.