Why a Missing Commit Can Block Unrelated Payments: A Spring Transaction Mystery

Incident Overview

A production outage occurred where payment requests reported success to users, but no rows were inserted into the order table. Occasionally, attempts to modify an order triggered lock‑timeout errors. A DBA discovered several long‑running transactions that never committed, holding locks on rows in the order table.

Abnormal Symptoms

Business code finishes without exception and logs show normal execution.

Log entries indicate commit() was called but the data is absent from the database.

Intermittent success : most requests fail, but a few manage to insert the order correctly.

Emergency Mitigation

The quickest way to restore the service was to restart the application, which forced the connection pool to discard the stale connections. After the restart the payment API worked again, but the underlying bug remained.

Root‑Cause Code

The offending service opened a transaction manually, performed an insert, and returned early when a special condition was met, forgetting to commit.

@Service
public class SomeService {
    public void handleSpecialCase() {
        // start transaction
        sqlSession.connection.setAutoCommit(false);
        mapper.insert(data);
        // special branch forgets commit!
        if (specialCondition) {
            return; // missing commit
        }
        sqlSession.commit();
    }
}

Adding a proper commit() inside the if block and surrounding the whole flow with try/catch to guarantee rollback on error fixes the problem.

@Service
public class SomeService {
    public void handleSpecialCase() {
        try {
            sqlSession.connection.setAutoCommit(false);
            mapper.insert(data);
            if (specialCondition) {
                sqlSession.commit(); // added commit
                return;
            }
            sqlSession.commit();
        } catch (Exception e) {
            sqlSession.rollback();
            throw e;
        }
    }
}

Post‑mortem Analysis

Transaction acquisition in Spring

Spring’s DataSourceTransactionManager obtains the current transaction via getTransaction(). The method first calls doGetTransaction() to retrieve a ConnectionHolder from TransactionSynchronizationManager:

protected Object doGetTransaction() {
    DataSourceTransactionObject txObject = new DataSourceTransactionObject();
    txObject.setSavepointAllowed(this.isNestedTransactionAllowed());
    ConnectionHolder conHolder = (ConnectionHolder) TransactionSynchronizationManager.getResource(this.obtainDataSource());
    txObject.setConnectionHolder(conHolder, false);
    return txObject;
}

If a previous transaction never called commit() or rollback(), the ConnectionHolder remains marked as active ( isTransactionActive() == true). Subsequent calls to getTransaction() therefore treat the connection as part of an existing transaction:

protected boolean isExistingTransaction(Object transaction) {
    DataSourceTransactionObject txObject = (DataSourceTransactionObject) transaction;
    return txObject.hasConnectionHolder() && txObject.getConnectionHolder().isTransactionActive();
}

How the polluted connection propagates

When SomeService.handleSpecialCase() returns early, the ConnectionHolder stays in the manager with its transaction flag still true. The next service (e.g., PaymentService.createOrder()) obtains the same connection, isExistingTransaction() returns true, and Spring joins the stale transaction instead of creating a new one.

private void processCommit(DefaultTransactionStatus status) throws TransactionException {
    try {
        prepareForCommit(status);
        triggerBeforeCommit(status);
        triggerBeforeCompletion(status);
        if (status.hasSavepoint()) {
            status.releaseHeldSavepoint();
        } else if (status.isNewTransaction()) {
            // real DB commit happens only for a new transaction
            doCommit(status);
        }
    } finally {
        cleanupAfterCompletion(status);
    }
}

Because the joined transaction is not a new one ( status.isNewTransaction() == false), doCommit() is never invoked and the underlying JDBC connection.commit() never runs. The data therefore never reaches the database.

Why the bug was intermittent

TransactionSynchronizationManager

stores resources in a ThreadLocal. If a request is processed by a thread that has a clean ConnectionHolder, the operation succeeds. If the thread reuses a polluted connection, the request fails. This thread‑pool scheduling makes the issue hard to reproduce.

Preventive Measures

1. Connection‑pool health checks

spring:
  datasource:
    hikari:
      connection-test-query: SELECT 1
      validation-timeout: 3000
      connection-init-sql: SET autocommit=1

The connection-test-query validates a connection before it is handed out, and connection-init-sql resets the session state, preventing a polluted connection from being reused.

2. Monitoring long‑running transactions

-- Find InnoDB transactions running longer than 30 seconds
SELECT *
FROM information_schema.innodb_trx
WHERE TIME_TO_SEC(TIMEDIFF(NOW(), trx_started)) > 30;

Set up alerts on the above query so that any transaction exceeding the threshold triggers a notification.

Key Takeaways

Connection pools reuse physical connections; a connection left in a transactional state can corrupt unrelated business logic.

When managing transactions manually, always place commit() in the try block’s final step and rollback() in the catch block.

Application logs may appear normal while the database layer hides problems; monitor slow queries, long‑running transactions, lock waits, and connection counts.

Debugging with breakpoints on getTransaction() and isExistingTransaction() is essential for uncovering hidden transaction‑propagation bugs.