Why Does a DB Connection Pool Exhaust in 2 Hours at <30 QPS? Slow SQL Isn’t to Blame
Even with less than 30 QPS and normal DB CPU, a connection pool can run out in two hours because connections are held too long—often due to leaks, long‑running transactions, or async misuse—so the root cause must be traced back to the application code rather than just slow SQL.
Commonly mis‑interpreted signals of pool exhaustion
ActiveConnections ≈ maximumPoolSize (e.g., maximumPoolSize=20)
IdleConnections drops to 0
ThreadsAwaitingConnection climbs continuously
Database CPU stays stable and slow‑SQL count does not increase noticeably
Application logs show "Connection is not available" or acquisition timeout
HikariCP defines ActiveConnections as connections currently held by the application, not the ones actively executing SQL. A connection may be held for seconds or minutes after the SQL finishes, causing the pool to die even under low load.
Low QPS can still saturate the pool
The pool measures concurrent usage, not request count. When the system is stable and leak‑free:
Concurrent connection demand ≈ request rate × connection hold timeExample: 30 QPS with each request holding a connection for 1 s yields a theoretical concurrency of 30, which exceeds a pool of size 20.
A slow leak can be worse: a piece of code leaks one connection every few minutes; after two hours the pool of 20 connections is empty. The typical progression is:
Active connections slowly rise
Idle connections slowly fall
Occasionally acquiring a connection becomes slow
Waiting threads increase
Requests get stuck acquiring connections
Before asking whether traffic is high, ask two basic questions:
After acquiring a connection, is it always closed/returned?
How long does the business logic hold the connection before returning it?
Diagnostic order: confirm pool saturation, then split “borrow slow” vs “return slow”
First confirm that connections are really unavailable. HikariCP suggests checking these metrics: ActiveConnections / IdleConnections /
TotalConnections ThreadsAwaitingConnection connection.acquire(acquisition time) connection.usage (hold duration)
If active connections are near the limit, idle is 0, and waiting threads keep rising, the bottleneck is “cannot get a connection”, not a slow endpoint.
Then determine whether borrowing is slow or returning is slow. A slow borrow only indicates the pool is empty; the root cause is usually earlier: a missing or delayed close().
Combine pool metrics with trace analysis to find:
Which interface has abnormal transaction duration
Whether the transaction contains RPC, uploads, or third‑party HTTP calls
If the DB session is an idle transaction
If leak logs repeatedly point to the same call stack
If an async task holds a JDBC connection for a long time leakDetectionThreshold can capture the stack of connections held too long; it only reports, it does not recycle.
Three distinct root causes
Connection not closed – common with hand‑written JDBC; exception paths may skip close(). Leak logs often show the same stack.
Long transaction – SQL may be fast, but the transaction encloses remote calls, heavy computation, or lock waiting, increasing both transaction and connection hold time.
Async misuse of connections – passing a Connection, Hibernate Session, or a transactional context to an async task is risky because Spring binds the transaction to the current thread; async threads do not inherit it.
Typical leak example
Connection conn = dataSource.getConnection();
PreparedStatement ps = conn.prepareStatement(sql);
ResultSet rs = ps.executeQuery();
if (!rs.next()) {
return null; // conn, ps, rs are not closed
}The correct approach follows the “who opens, who closes” rule within the same scope:
public User findById(long id) {
String sql = "select id, name from user where id = ?";
try (Connection conn = dataSource.getConnection();
PreparedStatement ps = conn.prepareStatement(sql)) {
ps.setLong(1, id);
try (ResultSet rs = ps.executeQuery()) {
if (!rs.next()) {
return null;
}
return new User(rs.getLong("id"), rs.getString("name"));
}
} catch (SQLException e) {
throw new IllegalStateException("query user failed", e);
}
}Framework expectations: JdbcTemplate – usually closes JDBC resources for you.
MyBatis + SqlSessionTemplate /Mapper – transactions are managed by Spring.
Native SqlSession – must be closed explicitly.
Container‑managed JPA + Spring transaction – EntityManager follows the transaction; manually created ones must be closed.
Mitigation: stop the bleed, fix the root, add safeguards
Stop the bleed – take the offending interface offline, apply rate limiting, or restart the service to clear the pool (restart only delays the problem).
Fix the root – ensure every acquisition path has a complete close path; avoid binding long‑running remote calls or heavy I/O inside a DB transaction. For messaging, use a transaction outbox with async delivery, retries, and idempotence; for cross‑service flows, consider saga or compensation patterns.
Add safeguards – monitor the following alerts together:
Active connections near the limit
Idle connections = 0
Waiting threads for connections keep rising
Connection acquisition time increases
Transaction duration exceeds business threshold
Leak detection logs appear
Slow SQL remains worth investigating when the system experiences high concurrency, DB CPU spikes, or execution time growth.
In summary, a connection pool reflects the application’s resource lifecycle, not database performance.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
