Why @Transactional Can Trigger Production Outages: Lessons from a Long‑Transaction Failure
The author recounts a real‑world incident where a Spring @Transactional annotation caused database connection exhaustion, deadlocks, and a production outage, explains how long‑running transactions arise from holding a single connection during costly operations, and offers practical ways to avoid them by splitting methods or using programmatic transactions.
Spring makes transaction management easy: adding @Transactional to a method lets the framework open, commit, or roll back a transaction automatically. Many developers equate the annotation with a database transaction and apply it to any method that touches the database.
In a 2019 internal reimbursement project, the author used @Transactional(rollbackFor = Exception.class) on a save() method that (1) called a workflow engine via HTTP, (2) converted a DTO, (3) saved the main record, and (4) saved detail records. The method looked simple and elegant, but the assumption that the annotation guarantees atomicity was wrong.
During a year‑end rush, many employees submitted reimbursement requests while the workflow engine was being hardened. The system experienced a spike in traffic, and the following symptoms appeared:
Database monitoring alarms about insufficient connections and massive deadlocks.
Log entries showing CannotGetJdbcConnectionException and time‑outs when calling the workflow service.
The connection pool became fully occupied, leading to repeated outages despite attempts to kill deadlocked processes or restart services.
Log analysis pinpointed the save() method as the root cause. The @Transactional annotation creates a single JDBC connection for the entire method via Spring AOP. When the method performs time‑consuming operations such as remote HTTP calls, the connection remains held for the whole duration, exhausting the pool and causing deadlocks – a classic long‑transaction problem.
The article defines a long transaction as a transaction that runs for a long time without committing, also called a “big transaction.” Its typical harms include:
Connection‑pool exhaustion, preventing other requests from obtaining a connection.
Increased likelihood of database deadlocks.
Long rollback times.
Greater replication lag in master‑slave setups.
To avoid long transactions, the author recommends reducing transaction granularity: split the method so that only the truly atomic database writes are wrapped in a transaction, while expensive operations (e.g., HTTP calls, validation) remain outside.
Spring supports two transaction styles:
Declarative transaction – using @Transactional. It is simple but ties the transaction scope to the whole method, limiting fine‑grained control.
Programmatic transaction – using TransactionTemplate or the low‑level API to start, commit, and roll back manually, allowing precise control over the transaction boundaries.
Example of programmatic transaction:
@Autowired
private TransactionTemplate transactionTemplate;
public void save(RequestBill requestBill) {
transactionTemplate.execute(status -> {
requestBillDao.save(requestBill);
requestDetailDao.save(requestBill.getDetail());
return Boolean.TRUE;
});
}The simplest way to prevent long transactions is to avoid placing @Transactional on methods that contain non‑database work. Instead, extract the database‑only part into a separate method and annotate only that method.
When developers still want to keep @Transactional, two practical solutions are offered:
Move the transactional method to another Spring bean (e.g., a manager layer) and inject it, ensuring that AOP proxies are applied.
Enable proxy exposure with @EnableAspectJAutoProxy(exposeProxy = true) and obtain the current proxy via AopContext.currentProxy() before invoking the transactional method.
Common scenarios where @Transactional silently fails are listed:
Annotation placed on a non‑public method. Incorrect propagation attribute. Incorrect rollbackFor attribute. Self‑invocation within the same class. Exceptions caught and not re‑thrown.
Correct method splitting examples:
@Service
public class OrderService {
public void createOrder(OrderCreateDTO dto) {
query();
validate();
saveData(dto);
}
@Transactional(rollbackFor = Throwable.class)
public void saveData(OrderCreateDTO dto) {
orderDao.insert(dto);
}
}And the proxy‑based approach:
@EnableAspectJAutoProxy(exposeProxy = true)
@SpringBootApplication
public class SpringBootApplication {}
public void createOrder(OrderCreateDTO dto) {
OrderService proxy = (OrderService) AopContext.currentProxy();
proxy.saveData(dto);
}In conclusion, while @Transactional is convenient, careless use can easily create long‑transaction problems that exhaust connections and cause deadlocks. For complex business logic, the author advises using programmatic transactions or carefully splitting methods according to the two patterns above.
Architect's Journey
E‑commerce, SaaS, AI architect; DDD enthusiast; SKILL enthusiast
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
