Why Did a Missing Commit Crash Our Payment Service? A Spring Transaction Deep Dive

Incident Overview

Last week the payment system went down: orders were reported as successful, but no rows appeared in the order table, and occasional lock‑timeout errors surfaced. A DBA saw several transactions that never committed, locking rows in the order table.

Abnormal Phenomena

Business code finishes normally with no errors and logs look fine.

Log shows commit called but the database contains no data.

Most requests fail while a few succeed sporadically.

Emergency Handling

The fastest way to restore service was to restart the application, which forced all connections to close and the payment flow worked again. However, the underlying bug remained.

@Service
public class SomeService {
    public void handleSpecialCase() {
        // open transaction
        sqlSession.connection.setAutoCommit(false);
        mapper.insert(data);
        // special branch forgets commit!
        if (specialCondition) {
            return; // missing commit
        }
        sqlSession.commit();
    }
}

The special branch returns before commit is executed, leaving the transaction open.

Quick Fix

@Service
public class SomeService {
    public void handleSpecialCase() {
        try {
            sqlSession.connection.setAutoCommit(false);
            mapper.insert(data);
            if (specialCondition) {
                sqlSession.commit(); // added commit
                return;
            }
            sqlSession.commit();
        } catch (Exception e) {
            sqlSession.rollback();
            throw e;
        }
    }
}

After adding the missing commit and redeploying, the issue disappeared.

Post‑mortem Analysis

Why did the missing commit affect other unrelated payment requests? The answer lies in Spring's transaction management internals.

Debugging getTransaction

public final TransactionStatus getTransaction(@Nullable TransactionDefinition definition) throws TransactionException {
    TransactionDefinition def = (definition != null ? definition : TransactionDefinition.withDefaults());
    Object transaction = doGetTransaction();
    boolean debugEnabled = logger.isDebugEnabled();
    if (isExistingTransaction(transaction)) {
        return handleExistingTransaction(def, transaction, debugEnabled);
    }
    // create new transaction logic …
}

The method first obtains the current transaction object via doGetTransaction. If it detects an existing transaction, it reuses it.

doGetTransaction Reuses Connections

protected Object doGetTransaction() {
    DataSourceTransactionObject txObject = new DataSourceTransactionObject();
    txObject.setSavepointAllowed(this.isNestedTransactionAllowed());
    ConnectionHolder conHolder = (ConnectionHolder) TransactionSynchronizationManager.getResource(this.obtainDataSource());
    txObject.setConnectionHolder(conHolder, false);
    return txObject;
}

TransactionSynchronizationManager.getResource

fetches a connection from the pool. If the connection was left in a transaction‑active state by a previous request, the ConnectionHolder carries that state forward.

isExistingTransaction Logic

protected boolean isExistingTransaction(Object transaction) {
    DataSourceTransactionObject txObject = (DataSourceTransactionObject) transaction;
    return txObject.hasConnectionHolder() && txObject.getConnectionHolder().isTransactionActive();
}

If the connection’s isTransactionActive() flag is still true, Spring treats it as an existing transaction and will not start a new one.

How a Polluted Connection Propagates

When SomeService.handleSpecialCase() returns early, the transaction remains active in the ConnectionHolder. The connection is returned to the pool via TransactionSynchronizationManager without resetting the flag. The next request, e.g. PaymentService.createOrder(), obtains this polluted connection, isExistingTransaction returns true, and the request joins the stale transaction. Because the request is not the transaction initiator, processCommit sees status.isNewTransaction() == false and skips the actual connection.commit(), so the order never reaches the database. SomeService.handleSpecialCase() skips commit.

The method ends, leaving ConnectionHolder.isTransactionActive() true. PaymentService.createOrder() calls doGetTransaction() and receives the same ConnectionHolder. isExistingTransaction evaluates to true, so Spring joins the existing transaction.

During commit, processCommit finds status.isNewTransaction() == false and does not execute doCommit.

The data stays uncommitted, and the polluted connection stays in the pool, affecting subsequent requests.

Why It Sometimes Succeeds

Spring stores TransactionSynchronizationManager in a ThreadLocal. If a request is handled by a thread that happens to pick a clean connection (no active flag), the transaction proceeds normally. If the thread picks the polluted connection, the bug appears. This explains the non‑deterministic success rate.

Preventive Measures

Connection Pool Health Checks

spring:
  datasource:
    hikari:
      connection-test-query: SELECT 1
      validation-timeout: 3000
      connection-init-sql: SET autocommit=1

The connection-init-sql resets the connection state each time it is borrowed, preventing a polluted connection from being reused.

Database‑Level Monitoring

-- Find transactions running longer than 30 seconds
SELECT *
FROM information_schema.innodb_trx
WHERE TIME_TO_SEC(TIMEDIFF(NOW(), trx_started)) > 30;

Configure alerts on long‑running transactions, lock waits, and connection counts to catch similar issues early.

Takeaways

Connection pools are not just performance optimisations; they can propagate transaction state bugs.

Always ensure commit and rollback are placed correctly, especially in early‑return branches.

Monitor the database layer (slow queries, long transactions, lock waits) in addition to application logs.

When facing mysterious issues, set breakpoints and step through the transaction management code; source reading alone may miss subtle state checks.