Why Your Service Returns 502 Errors Even When Its Logs Show No Issue

When a downstream service reports a 502 error but its own logs contain no 502 entries, the issue often lies in the gateway or proxy layer, typically nginx, which acts as a reverse proxy and load balancer.

HTTP status codes basics

Clients send HTTP requests to back‑end servers; the server replies with status codes: 200 for success, 4xx for client errors, and 5xx for server‑side problems. In many deployments the client does not talk directly to the back‑end but through a gateway (e.g., nginx).

How nginx returns 5xx codes

When nginx forwards a request to an upstream server, it opens a separate TCP connection. If the upstream server fails to respond correctly, nginx cannot forward a proper response and instead returns a 5xx code (commonly 502 Bad Gateway) to the client. The upstream itself may not log a 502 because it never generated one.

Common reasons for 502 Bad Gateway

Invalid response from upstream : The upstream sends a TCP RST or FIN packet, indicating an abrupt connection close.

Premature connection termination : The upstream server times out (e.g., WriteTimeout too short) and closes the connection before sending a full response. nginx then sees the reset and returns 502.

Server process crash : The application process exits or is killed (OOM, fatal error, explicit os.Exit()), leaving no listener on the expected port. The kernel replies with RST, which nginx translates to 502.

Misconfigured upstream IP/port : nginx’s upstream block points to an IP that no longer hosts the service. Requests are forwarded to a dead endpoint, producing RST and thus 502.

Diagnosing 502 issues

Since the backend may not log the error, start with the gateway:

Check nginx error logs for upstream connection failures or RST messages.

Verify the health of the upstream services: monitor CPU/memory for sudden drops, inspect process start times with ps -o lstart <pid>, and look for crash stack traces.

Ensure timeout settings (e.g., WriteTimeout in Go HTTP servers) are longer than the longest expected request processing time.

Confirm that the upstream IPs and ports in /etc/nginx/nginx.conf (or included files) match the currently running instances; automate registration if services are frequently redeployed.

Preventive measures

Increase appropriate timeout values to avoid premature connection resets.

Implement graceful shutdown and proper error handling in application code to avoid silent exits.

Use health checks and dynamic service discovery so nginx’s upstream list stays up‑to‑date.

Monitor for OOM kills and configure resource limits to prevent unexpected process termination.

Summary

HTTP status codes convey response outcomes; 5xx indicates server‑side problems.

nginx, as a reverse proxy, often generates 502 when upstreams send invalid TCP responses.

Typical 502 causes include premature timeouts, application crashes, and stale upstream configurations.

Diagnose by checking nginx logs, monitoring service health, and verifying timeout settings and upstream configurations.