Distributed Transactions: Theory, Seata AT/TCC, RocketMQ & Pitfalls Explained
This comprehensive guide covers distributed transaction theory (CAP/BASE), compares six solutions (2PC, TCC, SAGA, Seata AT, RocketMQ transaction messages, local message tables), provides runnable code for Seata AT, TCC, and RocketMQ patterns, and analyzes critical pitfalls like empty rollback, hanging, and idempotency with solutions.
One-Line Definition
Distributed transactions ensure atomicity across multiple independent microservices or databases: either all operations succeed or all roll back, achieving data consistency.
Why Distributed Transactions Exist
Monolithic applications : All operations run in a single database, relying on local ACID transactions for consistency.
Microservice architecture : A single business operation may span multiple services and databases, making local transactions ineffective.
Classic example : Order placement = deduct inventory + create order + deduct balance. If inventory deduction succeeds but order creation fails, inventory is lost. If order creation succeeds but balance deduction fails, the user gets the product for free.
Core Theoretical Foundations
CAP Theorem (Cornerstone of Distributed Systems)
CP : Sacrifice availability for strong consistency (e.g., ZooKeeper)
AP : Sacrifice strong consistency for availability (e.g., Eureka)
BASE Theory (CAP Compromise)
Basically Available : Basic availability
Soft State : Soft state
Eventually Consistent : Eventual consistency
Most internet companies adopt eventual consistency rather than strong consistency.
Mainstream Distributed Transaction Solutions Comparison
2PC (Two-Phase Commit)
Core Idea: Prepare phase + Commit phase
Consistency Level: Strong consistency
Performance: Very poor
Applicable Scenarios: Traditional cross-database
Representative Frameworks: MySQL XA
3PC (Three-Phase Commit)
Core Idea: 2PC improvement with timeout mechanism
Consistency Level: Strong consistency
Performance: Poor
Applicable Scenarios: Mostly theoretical
Representative Frameworks: No mainstream framework
TCC (Compensating Transaction)
Core Idea: Try-Confirm-Cancel three phases
Consistency Level: Eventual consistency
Performance: Good
Applicable Scenarios: Core business, high concurrency
Representative Frameworks: Seata-TCC
SAGA (Long Transaction)
Core Idea: Forward operations + reverse compensation
Consistency Level: Eventual consistency
Performance: Very good
Applicable Scenarios: Long processes, non-core business
Representative Frameworks: Seata-SAGA
Local Message Table
Core Idea: Local transaction + message queue
Consistency Level: Eventual consistency
Performance: Very good
Applicable Scenarios: Async scenarios, low consistency requirements
Representative Frameworks: RocketMQ Transaction Messages
Best-Effort Notification
Core Idea: Notify with maximum effort, retry on failure
Consistency Level: Eventual consistency
Performance: Best
Applicable Scenarios: Non-core business, tolerable inconsistency
Representative Frameworks: SMS/email notifications
Three Most Used Solutions in Big Tech
1. TCC Mode (Try-Confirm-Cancel)
Pros : High performance, strong flexibility, no lock blocking
Cons : High invasiveness, each service must implement three methods, large development effort
2. Local Message Table + Message Queue
Pros : Simple implementation, good performance, no distributed locks
Cons : Message table couples with business table, only suitable for async scenarios
3. Seata AT Mode (Alibaba Open Source, Most Mainstream)
Core Principle : Based on local transactions + two-phase commit, automatically generates rollback logs, zero intrusion to business code .
Pros : Extremely low development cost, good performance, backed by Alibaba
Cons : Only supports relational databases, some lock contention
Real Project Usage Experience
Core transaction chain : Use Seata AT mode to ensure consistency across ordering, payment, inventory
Non-core async scenarios : Use RocketMQ transaction messages, e.g., points distribution, SMS notifications
High-concurrency flash sale : Avoid distributed transactions; adopt "eventual consistency + compensation" — deduct inventory first, create order asynchronously, roll back inventory on failure
Summary
No silver bullet for distributed transactions; choose based on business scenario and consistency requirements :
Extreme consistency requirements → TCC
Pursue development efficiency → Seata AT
Async non-core scenarios → Local message table
Can tolerate eventual consistency → Prefer eventual consistency solutions
Remember: If local transactions can solve it, never use distributed transactions ! Distributed transactions are a last resort.
Core Code Implementations
1. Seata AT Mode (Zero Intrusion Preferred | Alibaba Standard)
Technical Highlight : Single annotation implements distributed transactions, zero business code intrusion, automatic rollback log generation.
/**
* Order Service - Global Transaction Entry
* Core Highlight: @GlobalTransactional annotation starts global transaction
* No need to modify any business logic, Seata automatically proxies all DB operations
*/
@Service
public class OrderServiceImpl implements OrderService {
@Autowired
private OrderMapper orderMapper;
@Autowired
private StockFeignClient stockFeignClient;
@Autowired
private AccountFeignClient accountFeignClient;
@Override
@GlobalTransactional(rollbackFor = Exception.class)
public Long createOrder(Long productId, Integer count, Long userId) {
// 1. Remote call to inventory service to deduct stock
stockFeignClient.deductStock(productId, count);
// 2. Remote call to account service to deduct balance
accountFeignClient.deductBalance(userId, productId.getPrice() * count);
// 3. Local order creation
Order order = new Order();
order.setProductId(productId);
order.setCount(count);
order.setUserId(userId);
order.setStatus(OrderStatus.PENDING);
orderMapper.insert(order);
return order.getId();
}
}2. TCC Mode (High-Concurrency Core Business | Best Performance)
Technical Highlight : Fully customizable transaction logic, no database lock blocking, supports non-relational databases.
/**
* Inventory Service TCC Interface Definition
* Core Highlight: @LocalTCC + @TwoPhaseBusinessAction auto-generates TCC proxy
*/
@LocalTCC
public interface StockTccService {
/**
* Try phase: Freeze inventory (resource reservation)
*/
@TwoPhaseBusinessAction(name = "stockTccService", commitMethod = "confirm", rollbackMethod = "cancel")
boolean tryDeductStock(
@BusinessActionContextParameter(paramName = "productId") Long productId,
@BusinessActionContextParameter(paramName = "count") Integer count);
/**
* Confirm phase: Actually deduct inventory (confirm execution)
*/
boolean confirm(BusinessActionContext context);
/**
* Cancel phase: Unfreeze inventory (rollback compensation)
*/
boolean cancel(BusinessActionContext context);
}/**
* TCC Implementation (Core Logic)
* Technical Highlight: Built-in transaction status table solves empty rollback, hanging, idempotency
*/
@Service
public class StockTccServiceImpl implements StockTccService {
@Autowired
private StockMapper stockMapper;
@Autowired
private TccTransactionLogMapper tccLogMapper;
@Override
@Transactional
public boolean tryDeductStock(Long productId, Integer count) {
String xid = RootContext.getXID();
// Anti-hanging: Check if Cancel already executed
if (tccLogMapper.exists(xid, "CANCEL")) {
return false;
}
// Idempotency: Check if Try already executed
if (tccLogMapper.exists(xid, "TRY")) {
return true;
}
// Freeze inventory
int rows = stockMapper.freezeStock(productId, count);
if (rows == 0) {
throw new RuntimeException("Insufficient inventory");
}
// Record transaction status
tccLogMapper.insert(xid, "TRY", "stock_service");
return true;
}
@Override
@Transactional
public boolean confirm(BusinessActionContext context) {
String xid = context.getXid();
// Idempotency check
if (tccLogMapper.exists(xid, "CONFIRM")) {
return true;
}
// Actually deduct frozen inventory
Long productId = (Long) context.getActionContext("productId");
Integer count = (Integer) context.getActionContext("count");
stockMapper.confirmDeduct(productId, count);
// Update transaction status
tccLogMapper.updateStatus(xid, "CONFIRM");
return true;
}
@Override
@Transactional
public boolean cancel(BusinessActionContext context) {
String xid = context.getXid();
// Anti-empty-rollback: Check if Try executed
if (!tccLogMapper.exists(xid, "TRY")) {
// Record Cancel status to prevent subsequent Try execution (hanging)
tccLogMapper.insert(xid, "CANCEL", "stock_service");
return true;
}
// Idempotency check
if (tccLogMapper.exists(xid, "CANCEL")) {
return true;
}
// Unfreeze inventory
Long productId = (Long) context.getActionContext("productId");
Integer count = (Integer) context.getActionContext("count");
stockMapper.unfreezeStock(productId, count);
// Update transaction status
tccLogMapper.updateStatus(xid, "CANCEL");
return true;
}
}3. RocketMQ Transaction Messages (Async Non-Core | Best Performance)
Technical Highlight : RocketMQ natively supports transaction messages, guaranteeing atomicity between local transaction and message sending.
/**
* Order Service Sends Transaction Message
* Core Highlight: sendMessageInTransaction atomically executes local transaction and message sending
*/
@Service
public class OrderServiceImpl implements OrderService {
@Autowired
private RocketMQTemplate rocketMQTemplate;
@Autowired
private OrderMapper orderMapper;
@Override
public Long createOrder(Long productId, Integer count, Long userId) {
Order order = new Order();
order.setProductId(productId);
order.setCount(count);
order.setUserId(userId);
order.setStatus(OrderStatus.PENDING);
// Send transaction message
rocketMQTemplate.sendMessageInTransaction(
"order_topic",
MessageBuilder.withPayload(order).build(),
order // Parameter passed to transaction listener
);
return order.getId();
}
}/**
* Transaction Message Listener (Core Logic)
* Technical Highlight: Auto-check mechanism solves unknown local transaction state
*/
@Component
@RocketMQTransactionListener
public class OrderTransactionListener implements RocketMQLocalTransactionListener {
@Autowired
private OrderMapper orderMapper;
@Autowired
private TransactionLogMapper transactionLogMapper;
@Override
@Transactional
public RocketMQLocalTransactionState executeLocalTransaction(Message msg, Object arg) {
try {
Order order = (Order) arg;
String transactionId = msg.getTransactionId();
// 1. Execute local transaction: Create order
orderMapper.insert(order);
// 2. Record transaction log (for check)
transactionLogMapper.insert(transactionId, "SUCCESS");
// Commit message, allow consumer to consume
return RocketMQLocalTransactionState.COMMIT;
} catch (Exception e) {
// Rollback local transaction, discard message
return RocketMQLocalTransactionState.ROLLBACK;
}
}
@Override
public RocketMQLocalTransactionState checkLocalTransaction(Message msg) {
String transactionId = msg.getTransactionId();
// Check local transaction execution status
if (transactionLogMapper.exists(transactionId, "SUCCESS")) {
return RocketMQLocalTransactionState.COMMIT;
} else {
return RocketMQLocalTransactionState.ROLLBACK;
}
}
}Core Technical Challenges & Solutions
Empty Rollback
Problem: Cancel phase executes but Try phase hasn't executed or failed (e.g., network timeout loses Try request)
Solution: 1. Each branch maintains transaction status table 2. Before Cancel, check if Try succeeded 3. If not, return directly and record Cancel status
Hanging Problem
Problem: Try phase times out, global transaction rolls back and executes Cancel, then Try request arrives, causing permanent resource freeze
Solution: 1. Record transaction status during Cancel 2. Before Try, check if Cancel already executed 3. If yes, reject Try execution
Idempotency Problem
Problem: Network retries cause same transaction to execute multiple times (e.g., TC retries Confirm/Cancel)
Solution: 1. Use global transaction ID (XID) as unique idempotency key 2. Check transaction status before execution 3. If already executed, return success directly
Data Consistency vs Performance Balance
Problem: Strong consistency (2PC) has terrible performance; eventual consistency has inconsistency window
Solution: 1. Core business uses Seata AT/TCC 2. Non-core uses local message table / best-effort notification 3. Scheduled reconciliation + manual compensation as safety net
Long Transaction Problem
Problem: Global transaction holds DB locks too long, causing concurrency performance drop
Solution: 1. Split long transactions into multiple short ones 2. Avoid remote calls and time-consuming ops in transactions 3. Async non-core logic
Distributed Lock Contention
Problem: Multiple transactions modifying same row cause lock wait timeout or deadlock
Solution: 1. Optimize DB indexes, use row-level locks 2. Control transaction granularity, shorten lock hold time 3. Use optimistic locking instead of pessimistic
TC Single Point Failure
Problem: Seata TC (Transaction Coordinator) down makes all global transactions unable to execute
Solution: 1. TC cluster deployment with Nacos registry 2. DB storage mode for transaction logs 3. Configure auto failover
Data Isolation Problem
Problem: Other transactions see intermediate state before global commit, causing dirty reads
Solution: 1. Seata AT uses global locks by default 2. Write isolation: Other transactions cannot modify same row before global commit 3. Read isolation: Default read uncommitted, configurable to read committed
Project Practical Experience Summary
Priority Principle : If local transactions can solve it, absolutely don't use distributed transactions; if eventual consistency can solve it, absolutely don't use strong consistency
Solution Selection :
90% of scenarios use Seata AT Mode (highest development efficiency)
High-concurrency core transactions use TCC Mode (best performance)
Async non-core scenarios use RocketMQ Transaction Messages (highest throughput)
Safety Net Mechanism : All distributed transaction solutions must combine scheduled reconciliation and manual compensation — the last line of defense for data consistency
Real Interview Simulation
Interviewer: "Introduce distributed transactions."
Candidate : Distributed transactions ensure atomicity across multiple independent databases/services — all succeed or all roll back, transitioning data from one correct state to another.
Typical example: User places order → Order service writes DB → Inventory service deducts stock → Points service adds points. If stock deduction fails but order already persisted, data becomes inconsistent — distributed transactions provide the safety net.
Must mention CAP Theorem : In distributed systems, Consistency, Availability, Partition Tolerance — max two simultaneously. Network partitions (P) are inevitable in production, so most systems choose AP or CP. Thus distributed transactions usually don't pursue strong consistency but Eventual Consistency (BASE) — Basically Available, Soft State, Eventually Consistent.
Common solutions comparison (simplified):
2PC/3PC : Strong consistency, Low throughput, Medium complexity, Legacy cross-DB XA
TCC : Eventual (business-controlled), High throughput, High complexity, Funds, red packets
Reliable Message Eventual Consistency : Eventual, High throughput, Medium complexity, Ordering, points chain
Saga : Eventual, High throughput, Medium-High complexity, Long processes, heterogeneous
Seata AT : Eventual, Medium-High throughput, Low complexity, Non-intrusive refactoring
① 2PC (Two-Phase Commit)
Coordinator asks all participants: "Can commit?" → Prepare phase. All Yes → send commit command; any No → full rollback.
✅ Strong consistency, but ❌ Synchronous blocking, coordinator single point, poor performance, rarely used in internet scenarios now.
② Reliable Message Eventual Consistency (Local Message Table + MQ)
Core idea: Upstream persists business and message first, then uses MQ to async notify downstream .
Typical example: RocketMQ Transaction Messages.
Send half-message (invisible to downstream)
Execute local transaction; success → commit half-message, failure → rollback
If local transaction commits but confirmation message lost? RocketMQ has check mechanism to actively query upstream for compensation
Downstream uses Unique Business ID for Idempotency — duplicate consumption won't cause errors
✅ High throughput, decoupled, ❌ Brief inconsistency but business-acceptable.
Interviewer: "Which solution does your project actually use? Why not TCC?"
Candidate : Our order-inventory-points scenario uses RocketMQ Transaction Messages + Local Message Table .
Business allows second-level delay, real-time consistency not required, so eventual consistency suffices.
TCC enables active rollback with stronger consistency, but every interface needs Try/Confirm/Cancel implementation — business intrusion too large , high development cost, plus must handle empty rollback, hanging headaches.
Trade-off: Message solution has best cost-performance, running two years in production with zero account imbalance issues ✅.
Interviewer: "If message sent but upstream local transaction rolls back, what happens?"
Candidate : RocketMQ Transaction Messages already prevent this 😄
Flow: Send half-message first — invisible to downstream. Local transaction commits → mark half-message deliverable; local transaction rolls back → half-message rolls back too, downstream never receives it.
So "message sent but local transaction rolled back" situation doesn't exist.
Interviewer: "Anything else to add?"
Candidate : Three bonus points 💡
Idempotency Design : Downstream must use business unique key (e.g., order number) as DB unique index to prevent duplicate consumption
Safety Net Scanning : If MQ has issues, scheduled task scans local message table for compensation, ensuring zero loss
Seata AT Mode : If you don't want to write much compensation code, consider it — non-intrusive auto-rollback, but accept global lock performance cost, suitable for low-concurrency scenarios
Final insight: No silver bullet for distributed transactions; if eventual consistency works, don't force strong consistency; choose based on business tolerance — that's the key to landing 👍
Interviewer: "Write core code for RocketMQ Transaction Messages, highlight technical points, then discuss technical challenges and solutions."
Core Code (Reliable Message Eventual Consistency)
Scenario: User places order, Order service sends transaction message, Inventory service consumes and deducts.
1. Order Service: Send Transaction Message (RocketMQ Transaction Messages)
// Order Service - Send half-message and execute local transaction
@Service
public class OrderService {
@Autowired
private RocketMQTemplate rocketMQTemplate;
@Transactional
public void createOrder(OrderDTO orderDTO) {
// 1. Generate unique business key (idempotency critical)
String orderId = generateOrderId();
// 2. Build message
Message<String> message = MessageBuilder.withPayload(orderId).build();
// 3. Send transaction message (Highlight: Half-message + Local Transaction + Check)
TransactionSendResult result = rocketMQTemplate.sendMessageInTransaction(
"order-topic",
message,
orderDTO // Business param, passed to local transaction executor
);
if (result.getSendStatus() != SendStatus.SEND_OK) {
throw new BusinessException("Order creation failed");
}
}
// Execute Local Transaction (RocketMQ Callback)
@RocketMQTransactionListener(txProducerGroup = "order-group")
public class OrderTransactionListener implements RocketMQLocalTransactionListener {
@Override
@Transactional
public RocketMQLocalTransactionState executeLocalTransaction(Message msg, Object arg) {
OrderDTO orderDTO = (OrderDTO) arg;
try {
// 1. Insert order table
orderMapper.insert(orderDTO.toOrder());
// 2. Insert local message table (for scheduled safety net)
localMessageMapper.insert(buildLocalMessage(msg));
// 3. Can pre-deduct inventory, freeze points here...
return RocketMQLocalTransactionState.COMMIT; // Local transaction success, commit half-message
} catch (Exception e) {
log.error("Local transaction execution failed", e);
return RocketMQLocalTransactionState.ROLLBACK; // Rollback half-message
}
}
@Override
public RocketMQLocalTransactionState checkLocalTransaction(Message msg) {
// Check: Query order table by business ID in message body to determine if persisted
String orderId = new String((byte[]) msg.getPayload());
Order order = orderMapper.selectByOrderId(orderId);
if (order != null) {
return RocketMQLocalTransactionState.COMMIT; // Already persisted,补提交
}
return RocketMQLocalTransactionState.ROLLBACK; // Not found,补回滚
}
}
}Technical Highlights:
Uses sendMessageInTransaction to implement Half-Message Mechanism , ensuring atomicity between local transaction and message delivery
Implements RocketMQLocalTransactionListener with executeLocalTransaction and checkLocalTransaction, perfectly handling network glitches and callback loss
Local transaction writes Order Table + Local Message Table simultaneously, leaving traces for scheduled safety net scanning
2. Inventory Service: Consume Message, Deduct Inventory (Idempotency Design)
// Inventory Service - Consume Order Message, Deduct Inventory
@Component
@RocketMQMessageListener(topic = "order-topic", consumerGroup = "stock-group")
public class StockDeductConsumer implements RocketMQListener<String> {
@Autowired
private StockService stockService;
@Override
public void onMessage(String orderId) {
// Idempotency Guarantee: Check if already processed by orderId
if (stockService.isAlreadyDeducted(orderId)) {
log.warn("Duplicate message, skipped processing: {}", orderId);
return;
}
// Start Inventory Deduction (Local Transaction)
try {
stockService.deductStock(orderId);
// Internal Implementation: Use Unique Index to Prevent Duplicate Deduction
// INSERT INTO stock_deduct_log (order_id) VALUES (#{orderId})
// If Duplicate Insert, Unique Index Throws Error, Catch Exception and Skip
} catch (DuplicateKeyException e) {
log.info("Idempotent interception, duplicate inventory deduction request: {}", orderId);
}
}
}Technical Highlights:
Unique Business Key Idempotency : Write deduction log table via order_id, DB Unique Index provides natural deduplication
Exception handling doesn't swallow errors, logs then continues consumption, ensuring no blocking of subsequent messages
3. Safety Net Scanning: Scheduled Task Compensation
// Scheduled Task - Scan Local Message Table, Compensate Unsuccessful Messages
@Component
public class LocalMessageCompensateJob {
@Autowired
private LocalMessageMapper localMessageMapper;
@Autowired
private RocketMQTemplate rocketMQTemplate;
@Scheduled(cron = "0 0/5 * * * ?")
public void compensate() {
// Query messages with status "Pending" and not updated for 2+ minutes
List<LocalMessage> messages = localMessageMapper.selectPendingMessages(2);
for (LocalMessage msg : messages) {
try {
// Resend to MQ
Message<String> mqMsg = MessageBuilder.withPayload(msg.getOrderId()).build();
rocketMQTemplate.syncSend("order-topic", mqMsg);
// Update local message status to Sent
localMessageMapper.updateStatus(msg.getId(), MessageStatus.SENT);
} catch (Exception e) {
log.error("Compensation send failed: {}", msg.getOrderId(), e);
}
}
}
}Technical Highlights:
Double Insurance : Even if RocketMQ check mechanism completely fails, scheduled scan guarantees eventual message delivery
Production can add distributed lock to prevent duplicate sending across multiple instances
Technical Challenges & Solutions for This Scenario
This Order → Inventory scenario hit many pits; summarized as Four Challenges 🚧
1. Reliable Message Delivery
Problem: Local transaction committed but message lost; or message sent but local transaction rolled back. How to guarantee one succeeds iff the other succeeds?
Solution: Adopt RocketMQ Transaction Messages (Half-Message + Check) . Send half-message invisible to downstream first. After local transaction commits, mark as deliverable. If local rolls back, half-message rolls back too — downstream never receives. Check mechanism solves half-message hanging from network timeouts.
2. Downstream Consumption Idempotency
Problem: MQ guarantees at-least-once delivery; duplicate consumption causes duplicate inventory deduction, overselling.
Solution: Establish Unique Identifier (Order Number) for each business operation. Downstream writes deduction log table using order_id as Unique Index . Duplicate insert catches DuplicateKeyException and skips.
3. Check Logic Design
Problem: When checkLocalTransaction is called, how to know if local transaction actually succeeded? No data found — not executed? Or executed but not yet persisted?
Solution: Check queries Order Table (or Local Message Table) by business key. If exists → transaction succeeded → return COMMIT; else → ROLLBACK. To prevent "executing" intermediate state, check method should add appropriate retry or wait.
4. Distributed Transaction Timeout
Problem: Order service local transaction succeeds, but Inventory service consumes message then stalls due to own fault (e.g., DB connection pool exhaustion), leaving user seeing order created but inventory unchanged.
Solution: ① Set reasonable consumption timeout and retry mechanism; ② Implement Dead Letter Queue in Inventory service to prevent blocking main queue; ③ Combine with Scheduled Local Message Table Scan for manual/auto compensation on long-unconsumed business (e.g., notify user, refund).
5. Empty Rollback (TCC Specific)
Problem: If using TCC directly, what if Cancel arrives before Try? Or Cancel called repeatedly?
Solution: In TCC scenarios must Allow Empty Rollback : Cancel returns success if no Try-reserved resources exist. Cancel operation must also be idempotent. Our architecture didn't use TCC, but this is a common interview follow-up — must be able to answer.
Interviewer : "Good, code highlights key points, challenge analysis thorough, especially idempotency and check explained clearly 👍"
Candidate : "Thanks. One more production tip: After launch, late-night slow queries caused local transaction execution to exceed MQ default check time, generating massive check calls. We raised local transaction timeout threshold and made check method read-only (no writes), as lightweight as possible — perfectly resolved 😄"
Interviewer : "Valuable experience. Interview done, wait for notification."
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
