How to Diagnose and Fix 502, 504, and Connection Reset Errors in Nginx

Problem Background

In production, 502, 504 and Connection Reset are the three most common error types in Nginx reverse‑proxy scenarios. Each error points to a different failure type:

502 Bad Gateway : backend does not respond.

504 Gateway Timeout : backend response is too slow.

Connection Reset : the connection is actively closed by the middle layer or the backend.

1. Distinguishing the Three Errors

1.1 Communication Chain

Client → Nginx (reverse proxy) → upstream (backend service)
        ↑                     ↑
Problem occurs at Client→Nginx   Problem occurs at Nginx→upstream

1.2 Error Feature Comparison

502

Log keyword: connect() failed Example: connect() failed (111: Connection refused) Direct cause: Nginx cannot connect to upstream.

502

Log keyword: no live upstreams Example: no live upstreams while connecting to upstream Direct cause: all upstreams are unavailable.

504

Log keyword: upstream timed out Example: upstream timed out (110: Connection timed out) Direct cause: upstream response timed out.

504

Log keyword: upstream prematurely closed Example: upstream prematurely closed connection Direct cause: upstream closed before completing response.

Connection Reset

Log keyword: recv() failed Example: recv() failed (104: Connection reset by peer) Direct cause: upstream actively reset the connection.

Connection Reset

Log keyword: Connection reset by peer Example: readv() failed (104: Connection reset by peer) Direct cause: Nginx or upstream actively closed.

1.3 Quick Identification Method

Do not rely solely on the browser error code. Use the following commands to inspect logs:

# 1. View actual status codes in access.log
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10
# 2. Correlate with upstream_status
tail -100 /var/log/nginx/access.log | grep "status=502" | head -5
# 3. Search error.log for key phrases
grep -E "connect() failed|upstream timed out|recv\(\) failed|Connection reset|no live upstreams" /var/log/nginx/error.log | tail -20

$status

is the response code Nginx returns to the client, while $upstream_status is the code returned by the upstream. They may differ (e.g., Nginx returns 504 while upstream actually returned 200).

2. Four‑Segment Chain Troubleshooting

Break the request path into four segments and verify each:

Segment 1: Client → Nginx network
Segment 2: Nginx itself
Segment 3: Nginx → upstream network
Segment 4: Upstream (backend service)

2.1 Local Comparison Method

Run curl on the Nginx host to rule out client‑to‑Nginx network issues:

# Request Nginx locally (bypass external network)
curl -sS -o /dev/null -w 'http_code=%{http_code} time_total=%{time_total}s time_connect=%{time_connect}s time_starttransfer=%{time_starttransfer}s
' http://127.0.0.1/health

If local curl succeeds but external access fails → problem in Segment 1.

If local curl also fails → problem in Segments 2‑4.

2.2 Direct Upstream Method

Bypass Nginx and curl the backend directly from the Nginx machine:

# Directly access upstream
curl -sS -o /dev/null -w 'http_code=%{http_code} time_total=%{time_total}s
' http://10.0.1.10:8080/health
# Test each upstream if there are multiple
for ip in 10.0.1.10 10.0.1.11 10.0.1.12; do
  echo -n "$ip: "
  curl -sS -o /dev/null -w 'code=%{http_code} total=%{time_total}s
' --connect-timeout 3 --max-time 5 http://$ip:8080/health
done

If Nginx fails and direct upstream also fails → Segment 4.

If Nginx fails but direct upstream works → Segment 2‑3 (Nginx config or Nginx‑upstream network).

3. Dedicated 502 Bad Gateway Troubleshooting

3.1 Log‑Based Diagnosis

# Find recent 502 entries in access.log
grep " 502 " /var/log/nginx/access.log | tail -5 | awk '{print $1,$4,$7}'
# Find related error.log entries
grep "connect() failed" /var/log/nginx/error.log | tail -10

3.2 Common Causes & Checks

Cause A: Backend process not started or crashed

# Check if backend process is alive
ps aux | grep -E "java|python|node|php-fpm" | grep -v grep
# Verify listening port
ss -lntp | grep 8080
# Look for OOM kills
dmesg -T | grep -i "oom\|killed" | tail -5

Cause B: PHP‑FPM pool exhausted

# PHP‑FPM status page (if enabled)
curl http://127.0.0.1/status
# Or check PHP‑FPM logs
tail -50 /var/log/php-fpm/www-error.log

Cause C: Firewall or security‑group blocking

# Test connectivity to upstream port
telnet 10.0.1.10 8080
nc -zv 10.0.1.10 8080
# Check iptables rules
iptables -L -n | grep 8080
# Verify cloud provider security group (outside of CLI)

Cause D: FastCGI buffer insufficient

# Search for buffer‑related errors
grep "upstream sent too big header" /var/log/nginx/error.log
# Fix by increasing buffers
location ~ \.php$ {
    fastcgi_buffer_size 32k;
    fastcgi_buffers 8 32k;
    fastcgi_busy_buffers_size 64k;
    ...
}

3.3 502 Checklist

# 1. Is backend process running?
ps aux | grep backend
# 2. Is port listening?
ss -lntp
# 3. Any firewall rules blocking?
iptables -L -n
# 4. Is proxy_pass correct?
grep "proxy_pass\|fastcgi_pass" /etc/nginx/conf.d/default.conf
# 5. Can upstream hostname resolve?
nslookup backend.example.com
# 6. Does backend expose health check?
curl -I http://127.0.0.1:8080/health

4. Dedicated 504 Gateway Timeout Troubleshooting

4.1 Timeout Configuration Overview

location /api/ {
    proxy_connect_timeout 5s;   # TCP connect timeout
    proxy_send_timeout 10s;    # Send request body timeout
    proxy_read_timeout 30s;    # Wait for upstream response (most common)
    proxy_pass http://backend;
}

The most frequent cause of 504 is an insufficient proxy_read_timeout. Nginx sends the request to upstream, but upstream does not return the full response header within the configured time.

Default values are 60 s for all three timeouts, which may be unreasonable in production.

4.2 Troubleshooting Steps

Step 1: Measure real upstream response time

# Use curl -w to capture timings
curl -sS -o /dev/null -w "
 time_namelookup=%{time_namelookup}s
 time_connect=%{time_connect}s
 time_starttransfer=%{time_starttransfer}s
 time_total=%{time_total}s
" http://10.0.1.10:8080/api/slow-endpoint

If time_starttransfer is large (e.g., > 30 s), increase proxy_read_timeout accordingly.

Step 2: Check error.log for timeout entries

grep "upstream timed out" /var/log/nginx/error.log | tail -5

Step 3: Split timeout settings per API type

# Quick API – 5 s timeout
location /api/quick/ { proxy_read_timeout 5s; proxy_pass http://backend; }
# Export – 120 s timeout
location /api/export/ { proxy_read_timeout 120s; proxy_pass http://backend; }
# Long‑poll / SSE – very long timeout, disable buffering
location /api/poll/ { proxy_read_timeout 3600s; proxy_buffering off; proxy_pass http://backend; }

4.3 Common Backend Root Causes for Slow Responses

Slow SQL – check database slow‑query log.

External dependency timeout – verify external API calls have their own timeout protection.

Thread‑pool queue – monitor backend thread‑pool metrics.

Deadlock – analyze thread dumps.

Full GC – review JVM GC logs.

Two mitigation directions: optimise backend performance (root cause) or enlarge proxy_read_timeout to keep the service available while optimisation is in progress.

5. Dedicated Connection Reset Troubleshooting

5.1 Nature of the Error

Connection Reset (104: Connection reset by peer) differs from 502/504: one side terminates the TCP connection with an RST packet before the normal four‑way handshake completes.

In Nginx this usually means:

Upstream actively closed – backend under heavy load closes connections.

Nginx actively closed – Nginx timeout or connection‑pool recycle.

Intermediate network device closed – firewall or load balancer idle‑timeout too short.

5.2 Troubleshooting Steps

Step 1: Identify which side reset the connection

# Upstream reset
grep "Connection reset by peer" /var/log/nginx/error.log
# Nginx‑initiated reset does not appear in error.log but clients see the reset

Step 2: Check upstream file‑descriptor (fd) usage and thread‑pool

# Inspect fd limits
cat /proc/$(pidof java)/limits | grep "open files"
# Count open fds
lsof -p $(pidof java) | wc -l
# Check TCP connection queues
ss -ant | grep -E 'SYN-RECV|TIME-WAIT' | wc -l
netstat -s | grep -i "listen overflow"

Step 3: Verify Nginx worker_connections

# Enable stub_status and query it
curl http://127.0.0.1/nginx_status
# Example output
Active connections: 65300
Reading: 0 Writing: 128 Waiting: 45

If active connections approach worker_connections × worker_processes, Nginx itself is saturated.

Step 4: Check TCP backlog overflow

# Listen queue overflow count
netstat -s | grep -i "listen"
# Current backlog size
ss -lntp | grep 80

If overflow is frequent, increase backlog and kernel parameters:

listen 8080 backlog=65535;
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_max_syn_backlog=65535

6. Nginx Configuration Optimisation Reference

6.1 Reasonable Timeout and Buffer Settings

upstream backend {
    least_conn;
    server 10.0.1.10:8080 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
    keepalive 128;
    keepalive_requests 10000;
    keepalive_timeout 60s;
}
server {
    listen 80 backlog=65535;
    server_name api.example.com;
    proxy_connect_timeout 5s;
    proxy_send_timeout 10s;
    proxy_read_timeout 30s;
    proxy_buffer_size 4k;
    proxy_buffers 8 4k;
    proxy_busy_buffers_size 8k;
    client_max_body_size 10m;
    client_body_buffer_size 128k;
    location /api/ {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass http://backend;
    }
}

6.2 Log Format Must Include Upstream Information

log_format main_ext '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" upstream_addr=$upstream_addr upstream_status=$upstream_status upstream_response_time=$upstream_response_time request_time=$request_time';
access_log /var/log/nginx/access.log main_ext;

The variables $upstream_addr, $upstream_status, $upstream_response_time and $request_time are essential for diagnosing 502/504/Connection Reset issues.

7. Quick Reference Scenarios

Occasional 502 – log shows connect() failed (111: Connection refused) → backend restart, port not ready. Check startup and health‑check.

Persistent 502 – log shows no live upstreams → all upstreams down. Verify all backend nodes.

Periodic 502 – log shows connect() failed at fixed intervals → system load spikes from cron jobs. Inspect crontab.

Partial 504 – log shows upstream timed out on specific API → analyse P99 latency of that endpoint.

All 504 – log shows upstream timed out everywhere → backend overload or DB connection pool exhaustion. Check CPU, connection pool, slow queries.

Intermittent Connection Reset – log shows recv() failed (104) → upstream fd shortage or thread‑pool saturation. Monitor fd and thread‑pool.

Batch Connection Reset – same code → backend OOM or crash. Review dmesg and backend logs.

502/504 alternating – both log keywords present → backend overload causing some requests to be rejected and others to timeout. Examine GC, thread‑pool, connection‑pool.

8. Production‑Environment Best Practices

Backup configuration before any change:

cp -a /etc/nginx /etc/nginx.$(date +%F_%H%M%S).bak

Test syntax and reload instead of full restart:

nginx -t
nginx -s reload

Configure per‑API timeouts based on SLA rather than a global timeout:

proxy_read_timeout 5s;   # quick API
proxy_read_timeout 120s; # export API
proxy_read_timeout 3600s; # long‑poll/SSE (disable buffering)

Close proxy_buffering for long‑polling or SSE to avoid delayed data delivery.

Limit proxy_next_upstream_tries to 2–3 to prevent retry storms on POST requests.

When 499 appears, it indicates the client closed the connection before Nginx responded (client‑side timeout).

9. Conclusion

Diagnosing 502, 504 and Connection Reset errors requires a data‑driven, segment‑by‑segment approach rather than blind restarts. The workflow is:

Log first : Identify the error keyword and the segment where it occurs.

Layered verification : Local curl → direct upstream → incremental checks.

Treat the symptom appropriately :

502 – verify backend availability and firewall.

504 – measure backend latency and adjust timeout settings.

Connection Reset – inspect backend file‑descriptors and thread‑pool.

All of this relies on comprehensive log fields ( $upstream_addr, $upstream_status, $upstream_response_time, $request_time). Without them, troubleshooting efficiency drops dramatically.