How to Diagnose and Fix 502 Bad Gateway Errors

HTTP Status Codes

Normal responses use 200. Client‑side errors are expressed with 4xx (e.g., 401 unauthorized, 404 not found). Server‑side problems are expressed with 5xx. In many cases a 5xx response is generated by a gateway or proxy rather than the origin server.

nginx as a reverse proxy

nginx sits between clients and a pool of backend servers. It accepts client connections, forwards each request to one of the backends, performs load‑balancing, and hides the actual server topology from the client.

Why nginx returns 5xx (502)

When a backend fails to respond, the TCP connection between nginx and the backend is broken. nginx detects the broken connection (typically a TCP RST or FIN) and returns a 502 Bad Gateway to the client. Because the backend never sees a complete request, it does not log a 502 entry.

Official definition (RFC 7231)

502 Bad Gateway
   The 502 (Bad Gateway) status code indicates that the server, while acting as a gateway or proxy, received an invalid response from an inbound server it accessed while attempting to fulfill the request.

In other words, the gateway received an invalid response from the upstream server.

Common causes of 502

RST – TCP reset packet

An RST aborts a TCP connection when a graceful four‑way termination is impossible. The application layer sees errors such as “connection reset” or “connection refused”.

Early connection closure (timeout)

If the backend closes the connection before nginx finishes sending the request, nginx receives an RST or FIN and returns 502. A typical scenario is a Go HTTP server with WriteTimeout=2s while the handler needs 5 seconds; the framework aborts the connection, nginx sees the FIN/RST, and the client receives 502. Increasing the timeout resolves the issue.

Backend process crash

When the backend process crashes, no process listens on the port. The kernel replies with an RST, which nginx forwards as 502. To verify a crash, check monitoring graphs for sudden CPU or memory drops and examine the process start time: ps -o lstart {pid} If the start time differs from the expected deployment time, the process likely restarted after a crash. Stack traces (if any) can reveal root causes such as writes to uninitialized memory or out‑of‑bounds array accesses.

Scenarios without a stack trace

Processes may be killed without a stack trace, e.g., the OOM killer terminates the process, or code calls log.Fatalln() which invokes os.Exit(). These cases also result in a 502 from nginx.

Misconfigured upstream IP

nginx forwards requests based on the upstream block in /etc/nginx/nginx.conf. If the listed IPs no longer host the service (e.g., after a deployment that removed an instance), the kernel returns RST and nginx returns 502. Example configuration:

upstream xiaobaidebug.top {
    server 10.14.12.19:9235 weight=2;
    server 10.14.16.13:8145 weight=5;
    server 10.14.12.133:9702 weight=8;
    server 10.14.11.15:7035 weight=10;
}

Updating the upstream configuration or using dynamic service registration avoids this problem.

Troubleshooting checklist

Check monitoring data for abrupt CPU or memory drops that indicate a backend crash.

Run ps -o lstart {pid} to compare the process start time with the expected deployment time.

If a crash is confirmed, examine any available stack trace to identify the root cause (e.g., null‑pointer dereference, array out‑of‑bounds).

If no crash is found, verify that the backend’s timeout settings (e.g., WriteTimeout) are sufficient for the longest request.

Inspect nginx logs for the upstream IP and port that was used; ensure the IP matches a running backend.

If the upstream IP is stale, update the nginx configuration or implement automatic service registration.