Skip to content
Who Crashed First? Troubleshooting 502 Errors Across Nginx, Thread Pools & DB Connection Pools
Operations 47 min read

Who Crashed First? Troubleshooting 502 Errors Across Nginx, Thread Pools & DB Connection Pools

This comprehensive guide walks through a systematic troubleshooting process for 502 Bad Gateway errors, detailing how to investigate Nginx upstream failures, application thread pool exhaustion, database connection pool saturation, slow queries, lock contention, and system resource bottlenecks with concrete commands and configuration examples.

Raymond Ops
Raymond Ops
Raymond Ops

Problem Background

A production 502 Bad Gateway error is one of the most feared incidents for ops and developers. The literal meaning is that the gateway (usually Nginx) received an invalid response from the upstream server (your application service). However, the real failure chain is far more complex: Nginx may report 502 because the application process crashed, the thread pool is exhausted, the database connection pool is full, a slow query hogs connections, an index is missing, lock waits occur, disk I/O is saturated, or an upstream third‑party service hangs. Every layer can become the bottleneck, yet all surface as the same 502 symptom.

Restarting services is only a band‑aid; root‑cause analysis requires preserving the evidence chain before any restart. Key questions: Does Nginx log show upstream timeout or connection refused? What is the application thread pool state? How many database connections are active and waiting? Are there slow queries, lock waits, or replication lag? Are CPU, memory, disk I/O, and network connections normal?

Applicable Scenarios

Frontend uses Nginx as reverse proxy

Backend: Java (Tomcat, Spring Boot, Dubbo), Python (uWSGI, Gunicorn, FastAPI), Go (Gin, Echo, stdlib)

Database: MySQL, PostgreSQL, or other relational DBs

Connection pools: HikariCP, Druid, c3p0, psycopg2, database/sql

Deployment: bare metal, VMs, or containers (Docker, Kubernetes)

Symptom: persistent or intermittent 502 errors; monitoring shows app running but responding abnormally

The troubleshooting mindset applies to any similar proxy → app → DB chain; just swap commands and configs.

Core Concepts

Nginx upstream Mechanism

Nginx defines upstream servers via the upstream block. On each request it picks a server per load‑balancing strategy, opens a TCP connection, sends the HTTP request, and waits for a response. If the upstream does not respond in time, refuses the connection, or returns an invalid HTTP response, Nginx returns 502.

Critical parameters: proxy_connect_timeout: connection establishment timeout proxy_send_timeout: request sending timeout proxy_read_timeout: response reading timeout

Upstream health checks: max_fails, fail_timeout Nginx error.log records the exact failure reason, e.g.:

upstream timed out (110: Connection timed out) while connecting to upstream
upstream prematurely closed connection while reading response header from upstream
connect() failed (111: Connection refused) while connecting to upstream

These logs are the first‑hand evidence.

Application Thread Pool Model

Most app servers (Tomcat, uWSGI, Gunicorn) use a thread/process pool: pre‑start a fixed number of workers; each request occupies one worker; if all workers are busy, new requests queue; if the queue fills, requests are rejected or hung.

Key Tomcat parameters: maxThreads: max worker threads (default 200) acceptCount: queue length (default 100) maxConnections: max concurrent connections (default 10000 for NIO)

If the thread pool is exhausted, Nginx times out waiting for a response and returns 502.

Database Connection Pool Principles

Applications borrow connections from a pool instead of creating new ones per request. Core HikariCP parameters: maximumPoolSize: max connections minimumIdle: min idle connections connectionTimeout: wait time for a connection maxLifetime: max connection lifetime idleTimeout: idle connection eviction time

When the pool is saturated, business logic blocks on getConnection() until timeout.

Slow Queries and Lock Waits

Even with free connections, a single long‑running SQL can tie up a connection, reducing effective pool capacity. Common causes:

Missing indexes → full table scans

Index invalidation (implicit type conversion, function on column, NULL checks)

Lock waits (InnoDB row locks, table locks, metadata locks)

Disk I/O bottleneck (low buffer pool hit ratio, frequent flushing)

Master‑slave replication lag (stale reads from replica)

System Resource Bottlenecks

High CPU → thread scheduling latency

Memory pressure → OOM Killer or frequent GC

Disk I/O saturation → read/write blocking

Network connection exhaustion → unable to establish new sockets

File descriptor exhaustion → cannot open files or sockets

Overall Troubleshooting Approach

Confirm failure scope: all or partial 502? continuous or intermittent? impact radius?

Check Nginx logs: upstream timeout or connection refused? exact error?

Verify app process: alive? port listening? resource usage?

Inspect app thread pool: active threads? queue length? any stuck threads?

Inspect DB connection pool: active connections? wait queue? long‑held connections?

Check DB state: slow queries? lock waits? replication healthy?

Check system resources: CPU, memory, disk I/O, network connections normal?

Identify root cause: combine logs, metrics, command output to find the earliest anomaly.

Fix and verify: apply targeted fix, then validate.

Prepare rollback: if fix is risky, have rollback steps ready.

Practical Steps

1. Confirm Failure Phenomenon

1.1 Check Monitoring Dashboard (Grafana, etc.)

HTTP status code distribution: 502 percentage?

Request success rate: sudden drop?

Response time: sudden spike?

QPS: traffic surge?

Record key timestamps and metrics for post‑mortem.

1.2 Review Alert History

Check Prometheus AlertManager, cloud monitoring, DingTalk/WeCom alerts: first alert time, content, multiple simultaneous alerts? Alert sequence often reflects the failure chain.

1.3 Gather User Feedback

Ask support/ops: which feature/page? regional differences? specific accounts or data triggers? Feedback can narrow scope.

2. Examine Nginx Logs

2.1 error.log

tail -n 500 /var/log/nginx/error.log | grep "502\|upstream"

Key errors:

upstream timed out (110: Connection timed out) while connecting to upstream

→ app process not started, port not listening, firewall block.

upstream timed out (110: Connection timed out) while reading response header from upstream

→ connection ok but app processing timeout (thread stuck or DB slow query).

connect() failed (111: Connection refused) while connecting to upstream

→ app process not listening on port (crashed or not started).

upstream prematurely closed connection while reading response header from upstream

→ app actively closed connection (crash or exception exit).

2.2 access.log

tail -n 500 /var/log/nginx/access.log | awk '$9 == 502 {print $0}'

URI distribution of 502s:

grep ' 502 ' /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

If 502s concentrate on a few endpoints, the issue may be specific business logic.

2.3 upstream Configuration

grep -A 10 "upstream" /etc/nginx/nginx.conf

Verify server IPs/ports, load‑balancing method, health‑check params ( max_fails, fail_timeout, keepalive).

2.4 Timeout Configuration

grep -E "proxy_connect_timeout|proxy_send_timeout|proxy_read_timeout" /etc/nginx/nginx.conf

Typical values:

proxy_connect_timeout 5s; proxy_send_timeout 60s; proxy_read_timeout 60s;

. Too short a proxy_read_timeout (e.g., 5s) can misclassify normal requests as timeouts.

3. Check Application Process State

3.1 Process Existence

ps aux | grep java

Confirm PID, start time, CPU/memory. If process missing, check app logs and system logs for OOM, crash, or manual kill.

3.2 Port Listening

netstat -tlnp | grep 8080
# or
ss -tlnp | grep 8080

Expected output: LISTEN 0 100 *:8080 *:* users:(("java",pid=12345,fd=123)). No output → app not listening (config error or start failure).

3.3 Port Connectivity Test

curl -I http://192.168.1.10:8080/health

Success → app process and port ok; problem may be Nginx config or network. Timeout/refused → issue on app side.

3.4 Application Logs

tail -n 500 /var/log/app/app.log | grep -E "ERROR|Exception|OutOfMemory"

Key errors: java.lang.OutOfMemoryError: Java heap space → heap too small or leak; tune JVM or analyze heap dump. java.net.SocketTimeoutException: Read timed out → downstream service or DB timeout. java.sql.SQLTimeoutException: Query timeout → DB query timeout.

org.apache.tomcat.util.threads.ThreadPoolExecutor: Pool exhausted

→ Tomcat thread pool exhausted.

4. Check Application Thread Pool

4.1 Capture Thread Stack

Java: jstack 12345 > /tmp/jstack_$(date +%Y%m%d%H%M%S).txt Python (uWSGI): uwsgi --connect-and-read /run/uwsgi/app.sock Go:

curl http://localhost:6060/debug/pprof/goroutine?debug=2 > /tmp/goroutine_$(date +%Y%m%d%H%M%S).txt

4.2 Analyze Thread Stack

Search for waiting for monitor entry or waiting on condition:

grep -A 10 "waiting for monitor entry\|waiting on condition" /tmp/jstack_*.txt | head -50

Many threads in WAITING / TIMED_WAITING on HikariPool.getConnection → DB side issue.

Many threads in BLOCKED on same object lock → code lock contention.

4.3 Tomcat Thread Pool Status

Enable Tomcat Manager in conf/tomcat-users.xml:

<user username="admin" password="your_password" roles="manager-gui"/>

Visit http://192.168.1.10:8080/manager/status; check currentThreadsBusy vs maxThreads. Near max → pool nearly exhausted.

4.4 uWSGI Process Status

uwsgi --connect-and-read /run/uwsgi/app.sock

Output shows workers accepting requests or busy. Many busy → requests processing slowly; investigate further.

5. Check Database Connection Pool

5.1 Pool Monitoring (HikariCP)

grep -i "hikari" /var/log/app/app.log | tail -50

Key log:

HikariPool-1 - Pool stats (total=10, active=10, idle=0, waiting=23)

→ pool full, 23 threads waiting.

5.2 Current DB Connections

SHOW PROCESSLIST;
# or
SELECT * FROM information_schema.PROCESSLIST WHERE COMMAND != 'Sleep';

Fields: Id, User, Host, db, Command (Query, Sleep, Killed), Time (seconds), State, Info (SQL text).

Many Sleep → connections not released (code leak or pool config). Many Query with high Time → slow queries.

5.3 Slow Query Log

tail -n 100 /var/log/mysql/slow.log

Example entry:

# Time: 2026-08-20T10:25:34.123456Z
# User@Host: app[app] @ [192.168.1.10]
# Query_time: 15.234567  Lock_time: 0.000123 Rows_sent: 1  Rows_examined: 1234567
SELECT * FROM orders WHERE user_id = 123456;

15 s execution, 1.23 M rows scanned, 1 row returned → missing index.

5.4 Lock Waits

SELECT * FROM information_schema.INNODB_LOCK_WAITS;

If rows exist, examine blocked and blocking transactions:

SELECT * FROM information_schema.INNODB_TRX WHERE trx_id = 1234567890;
SELECT * FROM information_schema.INNODB_TRX WHERE trx_id = 1234567891;

Kill blocking transaction if necessary: KILL 1234567891; Risk: rollback may affect data consistency; confirm impact first.

5.5 DB Performance Metrics

Buffer pool hit ratio:

SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool%';
# Hit ratio = (Innodb_buffer_pool_read_requests - Innodb_buffer_pool_reads) / Innodb_buffer_pool_read_requests

Below 95% → increase innodb_buffer_pool_size.

Disk I/O: iostat -x 1 10 Key: %util (100% = saturated), await (>20 ms = slow), r/s, w/s.

Top I/O process: iotop -o.

6. Check System Resources

6.1 CPU

top -bn1 | head -20
ps aux --sort=-%cpu | head -10

High wa (I/O wait) >30% → disk bottleneck.

6.2 Memory

free -h
dmesg | grep -i "out of memory"
grep -i "killed process" /var/log/messages

Low available (<10%) → OOM risk.

6.3 Disk I/O

iostat -x 1 10
iotop -o

Sustained %util 100% → upgrade disk or optimize I/O‑heavy ops.

6.4 Network Connections

ss -s
ss -tan | grep TIME_WAIT | wc -l
ss -tan | grep ESTAB | wc -l
ss -tan | grep :8080 | awk '{print $1}' | sort | uniq -c

Excessive TIME_WAIT (>10k) → tune kernel: sysctl -w net.ipv4.tcp_tw_reuse=1 (avoid tcp_tw_recycle in NAT).

6.5 File Descriptors

cat /proc/sys/fs/file-max
cat /proc/sys/fs/file-nr
ls /proc/12345/fd | wc -l
cat /proc/12345/limits | grep "open files"

Near limit → raise ulimit -n 65535 or edit /etc/security/limits.conf.

7. Root Cause Identification

Combine evidence to pinpoint the earliest anomaly. Six common scenarios:

Scenario 1: App Process Crashed

Evidence: Nginx connect() failed (111: Connection refused); ps aux shows no Java process; app log shows OutOfMemoryError or SegmentationFault.

Fix: increase JVM heap ( -Xmx4g -Xms4g), analyze heap dump ( jmap -dump:format=b,file=/tmp/heap.bin 12345), configure auto‑restart (systemd, supervisor, K8s).

Scenario 2: App Thread Pool Exhausted

Evidence: Nginx upstream timed out while reading response header; jstack shows many threads WAITING on HikariPool.getConnection; DB pool log shows active=10, idle=0, waiting=23.

Root cause: DB connection pool too small, dragging down thread pool.

Fix: increase maximumPoolSize (e.g., 20), optimize slow queries, increase app thread pool if resources allow.

Scenario 3: Slow Query

Evidence: app log SQLTimeoutException; slow log Query_time: 15.234567; SHOW PROCESSLIST many Query with high Time.

Root cause: missing index → full table scan.

Fix: add index ( CREATE INDEX idx_user_id ON orders(user_id);), optimize SQL, consider sharding.

Scenario 4: Lock Wait

Evidence: SHOW PROCESSLIST many Locked; INNODB_LOCK_WAITS shows waits; INNODB_TRX shows long‑running transaction holding lock.

Root cause: transaction holds lock too long.

Fix: kill blocker ( KILL 1234567891;), shorten transaction scope, adjust isolation level.

Scenario 5: Disk I/O Saturated

Evidence: iostat %util 100%; top %wa high; buffer pool hit ratio <95%.

Root cause: insufficient disk performance or too small buffer pool.

Fix: increase innodb_buffer_pool_size (e.g., 8G), upgrade to SSD/NVMe, optimize SQL to reduce I/O.

Scenario 6: Network Connection Exhaustion

Evidence: ss -s shows estab near system limit; dmesg shows TCP: too many orphaned sockets.

Root cause: connections not closed properly or short‑connection storm causing TIME_WAIT pile‑up.

Fix: ensure code closes connections, enable tcp_tw_reuse, use connection pools.

8. Fix and Verify

8.1 Pre‑Change Checklist

Backup current config files.

Validate changes in test environment.

Schedule during low‑traffic window.

Prepare rollback plan.

Notify stakeholders.

8.2 Example: Increase DB Connection Pool

cp /opt/app/config/application.yml /opt/app/config/application.yml.bak.$(date +%Y%m%d%H%M%S)

Edit application.yml:

spring:
  datasource:
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000

Restart: systemctl restart app. Verify log:

HikariPool-1 - Pool stats (total=20, active=5, idle=15, waiting=0)

.

8.3 Example: Add Database Index

mysqldump -u root -p --single-transaction --databases your_db > /backup/your_db_$(date +%Y%m%d%H%M%S).sql
SHOW CREATE TABLE orders;
CREATE INDEX idx_user_id ON orders(user_id);
SHOW INDEX FROM orders;
EXPLAIN SELECT * FROM orders WHERE user_id = 123456;

Expect type=ref, rows=1.

8.4 Post‑Fix Verification

Nginx access.log: 502 errors gone?

App logs: no more timeout exceptions?

Monitoring: success rate recovered?

DB pool: no waiting threads?

Observe at least 30 minutes to ensure no recurrence.

9. Rollback Plans

9.1 App Config Rollback

cp /opt/app/config/application.yml.bak.20260820102345 /opt/app/config/application.yml
systemctl restart app

9.2 Index Rollback

DROP INDEX idx_user_id ON orders;

Risk: query performance degrades; do in low‑peak.

9.3 Nginx Config Rollback

cp /etc/nginx/nginx.conf.bak.20260820102345 /etc/nginx/nginx.conf
nginx -t && nginx -s reload

9.4 Kernel Parameter Rollback

sysctl -w net.ipv4.tcp_tw_reuse=0
# or edit /etc/sysctl.conf then sysctl -p

Production Considerations

High‑Risk Operations (require low‑peak, notification)

Restart app / DB

Modify DB parameters

Add/drop indexes

Change kernel parameters

Kill DB transactions

Pre‑Operation Checklist

[ ] Config backed up?

[ ] Tested in staging?

[ ] Rollback plan ready?

[ ] Low‑peak window chosen?

[ ] Stakeholders notified?

[ ] Monitoring tools ready?

During Operation

Validate on single node first.

Verify config takes effect immediately.

Watch dashboards and logs continuously.

Roll back instantly on anomaly.

Post‑Operation Verification

App process healthy? Port listening? Health endpoint ok? Dashboards green? Logs clean? User complaints stopped? Observe 30+ minutes.

Post‑Mortem

Must include: timeline, root cause, actions taken, lessons learned, preventive measures, owner.

Monitoring & Alerting Recommendations

Nginx

502/503/504 count >10/min

Upstream response time >5s

Upstream failure count

Success rate <95%

QPS anomalies

Application (JVM)

Heap usage >80%

Full GC >5/hour

Active threads >180 (maxThreads=200)

API latency >3s

Database

Connections >150 (max_connections=200)

Slow queries >10/min

Lock waits >5

Buffer pool hit ratio <95%

Replication lag >10s

System

CPU >80%

Memory >90%

Disk I/O >80%

ESTABLISHED connections >10k

FD usage >80%

Log & Metric Observation Tips

Nginx Log Format

Include $request_time, $upstream_connect_time, $upstream_header_time, $upstream_response_time to pinpoint latency stage.

Application Log Format

Timestamp, level, thread, class/method, message, exception stack. Log request start/end, SQL execution time, remote call latency.

Database Slow Log

Enable with long_query_time=2, log_queries_not_using_indexes=1. Analyze Query_time, Lock_time, Rows_sent, Rows_examined.

Summary

502 troubleshooting is a systematic engineering effort across Nginx, application, database, and OS layers. Key principles:

Never blindly restart – restart is a tourniquet, not diagnosis; preserve evidence first.

Keep the evidence chain – logs, metrics, command outputs are artifacts for post‑mortem.

Layer‑by‑layer analysis – follow the request path: Nginx → app → DB → system.

Correlate multiple signals – never conclude from a single metric.

Verify the fix – monitor continuously after change.

Prepare rollback – every change can introduce new issues.

Conduct post‑mortem – document timeline, root cause, actions, lessons, improvements, owner.

Stability stems from deep system understanding and rigorous risk assessment. This guide aims to help you stay calm, locate the root cause quickly, and apply sound fixes when the next 502 strikes.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

operationsconnection poolMySQLincident responsenginxthread poollock contentionslow querysystem resources502 troubleshooting
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.