How a Silent Infinite Loop Drained Our Database Connection Pool

1. Incident Phenomenon

A batch task runs daily at 02:00 AM processing about 100,000 records. One morning the operations team received an alarm:

Database connections: 500/500 (full)
Application response time: > 5 seconds

The application stayed up, but every endpoint became slow because all requests were waiting for a database connection.

2. Investigation Process

Step 1: Check connection pool status

Used Arthas to query the Hikari pool MXBean:

ognl '@[email protected]()'

Output showed 50 active connections, 0 idle, and 150 threads awaiting a connection.

Step 2: Locate the time window

Examined logs and confirmed that after the batch task started at 02:00 AM, connections were never released.

tail -5000 logs/app.log | grep "Connection"

Step 3: Review batch task code

@Component
public class BatchTask {
    @Autowired
    private JdbcTemplate jdbcTemplate;

    @Scheduled(cron = "0 0 2 * * ?")
    public void processBatch() {
        List<Long> ids = getOrderIds();
        // Process 1000 records at a time
        for (Long id : ids) {
            // ❌ Potential infinite loop
            while (true) {
                try {
                    // Update order status
                    int updated = jdbcTemplate.update(
                        "UPDATE orders SET status = 1 WHERE id = ?", id);
                    if (updated > 0) {
                        break; // ✅ Successful exit
                    }
                    // ⚠️ If updated == 0, loop continues
                } catch (Exception e) {
                    log.error("Update failed", e);
                    // ❌ No break on exception, loop continues
                }
            }
        }
    }
}

The loop never breaks when updated == 0 or when an exception occurs, causing each thread to hold a DB connection indefinitely.

Step 4: Root‑cause analysis

int updated = 0; // No rows updated
if (updated > 0) {
    break; // ❌ Not executed
}
// ⚠️ Directly enters next iteration, repeats the same SQL
// Result: infinite loop

Consequently, every thread retains its connection until the process is killed, exhausting the pool.

3. Solution

3.1 Emergency fix

# 1. Restart the application to restore availability
systemctl restart app

# 2. Temporarily increase pool size (mitigation, not a cure)

3.2 Long‑term fix

Rewrite the batch logic with explicit retry limits and proper exit conditions:

@Scheduled(cron = "0 0 2 * * ?")
public void processBatch() {
    List<Long> ids = getOrderIds();
    for (Long id : ids) {
        int retryCount = 0;
        while (retryCount < 3) {
            try {
                int updated = jdbcTemplate.update(
                    "UPDATE orders SET status = 1 WHERE id = ?", id);
                if (updated > 0) {
                    break; // Success
                }
                // No rows updated – order does not exist
                log.warn("Order not found: {}", id);
                break;
            } catch (Exception e) {
                retryCount++;
                if (retryCount >= 3) {
                    log.error("Update failed after retries: {}", id, e);
                    failureService.record(id, e);
                    break;
                }
                log.warn("Update failed, retry {}: {}", retryCount, id);
                Thread.sleep(1000);
            }
        }
    }
}

Key principles for preventing infinite loops:

Infinite‑loop protection principles:
1. Every while(true) must have a clear exit condition.
2. Set a maximum number of iterations.
3. Decide whether to continue on exception.
4. Release resources when a DB operation fails.

4. Code Review Checklist

Batch code review checklist:
1. Does while(true) have an exit condition?
2. Is there a maximum loop count?
3. Does an exception cause the loop to continue?
4. Are resources released after a DB failure?
5. Is there a timeout mechanism?

5. Final Thoughts

A single unnoticed infinite loop can drain the entire connection pool and bring the application to a standstill. Always verify exit conditions in retry logic.