Postmortem: How a Synchronous DB Insert in a Signature Interceptor Exhausted the Connection Pool

On July 18, 2026, an alert showed "HikariPool-1 - Connection is not available, request timed out after 30000ms," indicating the database connection pool was exhausted. The microservice had not been redeployed recently, so the spike in traffic was suspected.

Background

The system receives order and refund data from a third‑party service. For each request, the inbound parameters are saved so that, if the synchronization later fails, the original payload can be replayed.

Investigation

The stack trace pointed to a signature‑verification interceptor that attempted to insert the request parameters into the database. The interceptor’s primary job is to parse the request, verify the signature, and allow the request to proceed; however, after a successful verification it also performed a synchronous insert.

Insert the inbound request into the database to retain the original message for later replay and compensation.

The core logic was sound, but the implementation was flawed: a non‑essential operation was placed inside a critical path.

Signature verification is a core process; persisting the request is non‑core. Mixing them makes the core vulnerable to failures in the auxiliary step.

Each incoming request caused the Tomcat worker thread to acquire a database connection during the interceptor phase. With a modest number of stores the pool could handle the load, but after opening many new stores the request volume surged. Tomcat’s default of 200 worker threads simultaneously competed for the limited pool, quickly exhausting it and causing 30‑second timeouts.

Cascade Impact

The connection pool is shared globally, so when the interceptor monopolized connections, other services—order sync, refund sync, and any feature needing database access—also failed.

A single non‑core synchronous insert thus drained the entire system’s database connections, cascading failures across all core business functions.

Emergency Fix

Recognizing that the interceptor should not perform synchronous DB work, the author changed the persistence to an asynchronous task. After verification, the request payload is submitted to a dedicated thread pool, allowing the Tomcat thread to return immediately without holding a connection.

Core code change:

// Before: synchronous save, blocks verification flow
accessRecordService.save(build);
// After: asynchronous save, does not block verification flow
accessRecordPool.execute(()->accessRecordService.save(build));

The auxiliary thread pool has 2 core threads, a maximum of 5 threads, and a queue size of 2000. The lightweight save operation fits this configuration. Failures in the async task are logged without affecting verification. After deploying the fix on July 18, the same alert did not reappear on July 19.

Better Long‑Term Solutions

While async execution removed the immediate blockage, it still relies on the same connection pool, so a pool exhaustion would cause record loss. Moreover, an in‑memory queue loses pending tasks on process restart.

Since the saved request is only needed for replay and compensation—both non‑real‑time scenarios—the author proposes decoupling collection from persistence:

Message‑queue decoupling: the interceptor publishes the payload to an MQ; a consumer batches inserts, benefiting from MQ durability and retry mechanisms.

Log‑file fallback: write the payload as JSON to an independent log file via an asynchronous appender, avoiding database connections entirely and providing persistence even if the process crashes.

Batch inserts: regardless of the persistence method, aggregate records and insert them in bulk to reduce connection usage.

Postmortem

Root cause: a synchronous database operation that belonged in a non‑core path filled the connection pool under increased traffic, causing a cascade of failures.

Broader lesson: developers must clearly distinguish core processes from auxiliary functions and ensure that non‑critical work does not block or compete for resources essential to core functionality.