Frequent Nginx 502/504 Errors? Understanding the Underlying Logic of Upstream Timeout Settings

Timeout points in the request lifecycle

The client‑to‑Nginx read/write timeout differs from the Nginx‑to‑upstream phases. proxy_connect_timeout controls connection establishment, proxy_send_timeout the interval between two write operations to the upstream, and proxy_read_timeout the interval between two read operations from the upstream (it is not the total request time). The upstream group parameters max_fails and fail_timeout affect passive removal but do not trigger active health checks.

Step‑by‑step troubleshooting

Confirm the effective Nginx version and compiled modules.

nginx -V 2>&1
nginx -v
systemctl status nginx --no-pager

Run a syntax check and dump the full effective configuration.

sudo nginx -t
sudo nginx -T > /var/tmp/nginx-effective.conf
rg -n 'proxy_(connect|send|read)_timeout|upstream|proxy_pass' /var/tmp/nginx-effective.conf

The -T output expands includes and may contain certificate paths; store it securely.

Distinguish 502, 504 and client disconnects.

awk '$9 ~ /^(502|504)$/ {print $4, $7, $9, $10, $11}' LOG_DIR/access.log | tail -n 100
rg -n 'upstream|timed out|connect\(\) failed|prematurely closed' LOG_DIR/error.log | tail -n 100

Field $9 in the access log holds the status code; the error log text determines the next step. Do not draw conclusions from status codes alone.

Add upstream timing fields to logs.

log_format upstream_timing '$remote_addr $request $status '
    'rt=$request_time uct=$upstream_connect_time '
    'uht=$upstream_header_time urt=$upstream_response_time '
    'ua="$upstream_addr" us="$upstream_status"';
access_log LOG_DIR/access-upstream.log upstream_timing;

Use request_time for total Nginx processing time and upstream_response_time for upstream latency; multiple retry addresses may produce comma‑separated values.

Check listening ports and local connectivity.

ss -lntp | rg ':(80|443|UPSTREAM_PORT)\b'
curl -sS -o /dev/null -w 'http=%{http_code} connect=%{time_connect} start=%{time_starttransfer} total=%{time_total}
' \
    http://UPSTREAM_ADDRESS/health

If the upstream uses host‑based routing, add -H 'Host: DOMAIN' to the curl command.

Inspect upstream configuration and failure thresholds.

upstream app_backend {
    least_conn;
    server 10.0.0.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=10s;
    keepalive 64;
}

max_fails is a passive failure counter; fail_timeout is both the statistics window and the temporary unavailability period. It does not replace health checks.

Define which passive failures trigger retries.

location /api/ {
    proxy_pass http://app_backend;
    proxy_next_upstream error timeout http_502 http_503 http_504;
    proxy_next_upstream_tries 2;
}

Enable retries only for idempotent requests; POST, payment, or resource‑creation calls without an idempotency key may cause duplicate writes.

Examine error‑log messages for connection failures.

rg -n 'connect\(\) failed' LOG_DIR/error.log | tail -n 50
ip route get UPSTREAM_IP
nc -vz -w 3 UPSTREAM_IP UPSTREAM_PORT

nc validates TCP connectivity only; it does not guarantee HTTP availability. As an alternative, use timeout 3 bash -c '</dev/tcp/UPSTREAM_IP/UPSTREAM_PORT>' (requires Bash).

Capture a real response header.

curl -sv --max-time 10 -H 'Host: DOMAIN' http://UPSTREAM_ADDRESS/PATH \
    -o /dev/null 2>&1 | sed -n '/^> /p;/^< /p'

Reproduce with the same URI, Host, authentication, and protocol; do not test only / .

Understand proxy_connect_timeout . It takes effect when connection queues or network black‑holes occur.

Distinguish send timeout from read timeout. Slow upstream receipt or large request bodies may trigger send timeout first; slow application processing usually appears as read timeout.

Set explicit limits for large requests.

client_max_body_size 20m;
client_body_timeout 15s;
client_body_buffer_size 128k;

Changing limits impacts upload services; verify API contracts, front‑end error messages, and direct‑to‑object‑store options before adjusting.

Control proxy buffering strategy.

location /stream/ {
    proxy_pass http://app_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_read_timeout 3600s;
}

Streaming, SSE and download endpoints should not use the default JSON buffering.

WebSocket requires upgrade headers.

location /ws/ {
    proxy_pass http://app_backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 3600s;
}

Missing upgrade headers often manifest as abnormal connection closure after the handshake.

Check upstream keepalive configuration.

location /api/ {
    proxy_pass http://app_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
}

Only writing keepalive without HTTP/1.1 usually does not achieve the expected reuse.

Inspect Nginx connections and file‑descriptor usage.

pid=$(cat /run/nginx.pid)
cat /proc/${pid}/limits | rg 'open files'
ls /proc/${pid}/fd | wc -l
ss -s

If the count approaches the FD limit, determine whether long connections, log files, upstream connections, or leaks are responsible before raising LimitNOFILE in systemd.

Check worker and system limits.

worker_processes auto;
worker_rlimit_nofile 65535;
events { worker_connections 4096; }

These are not recommended defaults. The theoretical connection capacity also depends on worker count, upstream connections, FD limits, and kernel constraints; determine values from monitoring and load‑testing.

Observe upstream queue and application resources.

curl -s http://UPSTREAM_ADDRESS/metrics | rg 'http.*(inflight|duration|requests)|process_open_fds' | head -n 80
ps -eo pid,ppid,%cpu,%mem,etime,cmd --sort=-%cpu | head -n 20

504 is usually a symptom; the root cause often lies in the upstream.

Correlate slow requests.

awk 'match($0,/rt=([0-9.]+)/,a) && a[1] > 10 {print}' LOG_DIR/access-upstream.log | tail -n 100

The awk depends on the custom log format defined earlier; slow requests only indicate duration, the root cause still needs upstream tracing, SQL slow‑log, or dependency metrics.

Validate database dependencies.

mysqladmin -h DB_ADDRESS -u CHECK_USER -p ping
mysql -h DB_ADDRESS -u CHECK_USER -p -e "SHOW GLOBAL STATUS LIKE 'Threads_running';"

This is not to make Nginx depend on MySQL directly; it is for troubleshooting critical upstream dependencies. Use a low‑privilege read‑only account.

Use clear site configuration with proper include relationships.

server {
    listen 80;
    server_name DOMAIN;
    location /api/ {
        proxy_pass http://app_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_connect_timeout 3s;
        proxy_read_timeout 30s;
    }
}

Check whether proxy_pass retains the URI prefix and trailing slash; verify with test URIs before changing.

Backup and replace configuration.

#!/usr/bin/env bash
set -euo pipefail
CONF="SITE_CONFIG"
STAMP=$(date +%F-%H%M%S)
sudo cp -a "${CONF}" "${CONF}.${STAMP}.bak"
sudo nginx -t
sudo systemctl reload nginx
curl -fsS -H 'Host: DOMAIN' http://127.0.0.1/HEALTH_PATH > /dev/null

Reload loads the new configuration and smoothly replaces workers. If health checks fail, roll back immediately instead of further tweaking timeouts.

Rollback configuration.

set -euo pipefail
sudo cp -a "SITE_CONFIG.TIMESTAMP.bak" "SITE_CONFIG"
sudo nginx -t
sudo systemctl reload nginx

Confirm the timestamp matches the current change to avoid overwriting others' updates. After rollback, verify error rates and key interfaces.

Gray‑release the change.

sudo nginx -t && sudo systemctl reload nginx
for n in 1 2 3 4 5; do
  curl -fsS -o /dev/null -w '%{http_code} %{time_total}
' \
      -H 'Host: DOMAIN' http://127.0.0.1/HEALTH_PATH
done

The loop is a local probe, not a load test. Observe entry error rate, upstream timing, and upstream load during the gray period.

Verify the effective configuration.

sudo nginx -T 2>/dev/null | sed -n '/upstream app_backend/,/^}/p'
sudo nginx -T 2>/dev/null | rg -n 'proxy_(connect|send|read)_timeout'

Do not treat editor content as evidence; include order of include statements and possible duplicate locations.

Build short‑term status‑code statistics.

awk '{count[$9]++} END {for (s in count) print s, count[s]}' LOG_DIR/access.log | sort -n

Align the statistics window with the change time. A drop in status count but a P99 increase may still indicate timeout‑induced latency shift.

Monitor error ratio with Prometheus.

sum(rate(nginx_http_requests_total{status=~"502|504"}[5m]))
/ clamp_min(sum(rate(nginx_http_requests_total[5m])), 1)

If the exporter does not expose nginx_http_requests_total , check the /metrics endpoint first; do not invent metric names.

Observe upstream latency quantiles.

histogram_quantile(0.99,
  sum by (le) (rate(http_server_request_duration_seconds_bucket[5m])))

The query depends on the application exposing a histogram; metric names, labels and units must match the actual exporter.

Check DNS resolution problems.

getent ahostsv4 UPSTREAM_DOMAIN
resolvectl query UPSTREAM_DOMAIN 2>/dev/null || true

When using containers, Kubernetes or cloud DNS, verify the resolution chain inside the Nginx network namespace.

Form an incident evidence package.

#!/usr/bin/env bash
set -euo pipefail
OUT="/var/tmp/nginx-incident-$(date +%F-%H%M%S)"
mkdir -p "${OUT}"
nginx -T > "${OUT}/nginx-effective.conf" 2>&1
ss -s > "${OUT}/socket-summary.txt"
 tail -n 500 LOG_DIR/error.log > "${OUT}/error-tail.log"
 tail -n 1000 LOG_DIR/access-upstream.log > "${OUT}/access-tail.log"
 tar -C "$(dirname "${OUT}")" -czf "${OUT}.tgz" "$(basename "${OUT}")"

The package may contain internal addresses, URIs or identity information; restrict access and store it according to security procedures. Conclusions must be supported by logs, metrics, configuration diffs, or upstream diagnostics: for 502 first check reachability and valid response; for 504 examine response intervals and upstream queue; adjust timeouts only after business limits are clear and the root cause is isolated.

Isolate long‑running tasks with a dedicated location.

location = /api/report/export {
    proxy_pass http://app_backend;
    proxy_connect_timeout 3s;
    proxy_send_timeout 30s;
    proxy_read_timeout 120s;
    proxy_next_upstream off;
}
location /api/ {
    proxy_pass http://app_backend;
    proxy_connect_timeout 3s;
    proxy_read_timeout 30s;
}

Export endpoints that are not idempotent should not have automatic retries. Verify exact URI matching (the = operator) during gray rollout.

Align client, load‑balancer and upstream timeouts.

curl -sS -o /dev/null -w 'code=%{http_code} connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total}
' \
    --max-time 35 -H 'Host: DOMAIN' http://PROXY_ADDRESS/PATH

This end‑to‑end probe must use the interface SLO for --max-time . Multi‑layer gateways, CDNs and cloud load balancers also have idle timeouts that need to be recorded in the change ticket.

Distinguish upstream header timeout from response‑body interruption.

rg -n 'upstream timed out|upstream prematurely closed connection|recv\(\) failed' LOG_DIR/error.log | tail -n 200

upstream prematurely closed connection usually indicates upstream process restart, crash or intentional disconnect; it should not be blindly turned into a 504. Check upstream process logs and deployment events.

Check connection state after change.

ss -tan '( sport = :80 or sport = :443 )' | awk 'NR>1 {state[$1]++} END {for (s in state) print s, state[s]}'
cat /proc/net/sockstat

Abnormal growth of TIME_WAIT, ESTAB or orphan states can indicate connection‑lifecycle problems. Compare statistics over identical windows before and after the change.

Common misconfiguration: increasing timeouts does not increase capacity

When the upstream already has thread‑pool queuing, changing proxy_read_timeout from 30 s to 300 s only keeps more client connections alive longer in Nginx. The increased concurrency consumes workers, file descriptors and upstream connections, potentially turning a single slow endpoint into a site‑wide latency problem. Define business‑level limits for each interface type (synchronous query, report export, file upload, streaming, asynchronous task) and avoid sharing a single location’s timeout policy.