Distributed Transaction Deep Dive: CAP/BASE, TCC/SAGA & 8 Production Faults

The previous 20 articles formally concluded the complete core series on Java online fault investigation and performance tuning, covering servers, JVM, databases, middleware, gateways, stress testing, and architecture safety nets.

In production microservice architectures, however, there remains a class of problems that are highly concealed, extremely damaging, and hardest to troubleshoot — distributed transaction issues — which are a capability gap for most engineers.

In monolithic projects, local database transactions easily achieve ACID consistency. After microservice decomposition, cross-service, cross-database, and cross-middleware scenarios break local transaction constraints, leading to data inconsistency, transaction suspension, half-transactions, rollback failures, and dirty data .

Many online P1 incidents — fund discrepancies, abnormal order states, inventory mismatches, missing ledgers — stem not from code bugs but from wrong distributed transaction selection, non-standard implementation, and missing exception safeguards .

This article, as advanced bonus episode 1 , dives deep into production distributed transaction scenarios, covering underlying theory, mainstream solutions, high-frequency faults, troubleshooting SOP, and root-cause optimization to fill the last core gap in microservice architecture, suitable for both production landing and senior interviews.

I. Core Fundamentals of Distributed Transactions (Eliminate Half-Understanding)

1. What Is a Distributed Transaction?

When a single business operation requires reads/writes across multiple databases, multiple microservices, and multiple middleware components , and all operations must either fully succeed or fully roll back to guarantee data consistency, such cross-node transactions are called distributed transactions.

Core pain point : After microservice decomposition, atomicity support from local transactions is lost, causing partial success and partial failure data inconsistency.

2. CAP Theorem (Cornerstone of Distributed Architecture)

A distributed system cannot simultaneously satisfy all three of CAP; it must trade off:

C (Consistency) : All nodes see the same data at the same time.

A (Availability) : System remains accessible and responsive.

P (Partition Tolerance) : System continues operating despite network partitions or node failures.

Production core conclusion : Distributed systems must guarantee P (network failures are the norm), so the choice is between CP and AP.

CP architecture: Sacrifice availability for strong consistency (finance, payment core scenarios).

AP architecture: Sacrifice strong consistency for high availability (e-commerce, general business scenarios).

3. BASE Theory (Core of Production Eventual Consistency)

Internet high-concurrency projects almost all follow BASE flexible transaction theory, abandoning strong consistency in pursuit of eventual consistency :

BA (Basically Available) : During failures, allow partial degradation or rate limiting to keep core business available.

S (Soft State) : Allow data to be temporarily inconsistent, existing in intermediate states.

E (Eventual Consistency) : Through asynchronous compensation, retries, and reconciliation, data eventually becomes fully consistent.

All production distributed transaction solutions are entirely based on BASE eventual consistency ; this is the foundational core for troubleshooting and optimizing transaction faults.

II. Four Mainstream Distributed Transaction Solutions (Production Selection + Applicable Scenarios)

Production boils down to four distributed transaction solutions; each maps to different fault characteristics and troubleshooting focuses. Wrong selection is the primary root cause of transaction anomalies.

1. 2PC/XA Strong Consistency Transaction

Core principle : Divided into prepare phase and commit phase; all participants ready then commit uniformly, any node failure triggers global rollback.

Pros : Strong consistency, no data deviation.

Fatal cons : Severe blocking, extremely poor performance, no high-concurrency support, node failures easily cause suspension.

Applicable scenarios : Traditional finance, low concurrency, strong consistency scenes; basically abandoned in internet high-concurrency projects .

2. Local Message Table (Most Widely Adopted, Simplest & Most Stable)

Core principle : Business table + message transaction table; local transaction guarantees business data and message record land together; background scheduled task scans unsent messages, retries delivery, updates status after consumption success.

Pros : No strong middleware dependency, simple and stable, high fault tolerance, fits vast majority of businesses.

Cons : Requires maintaining transaction table, slight code intrusion.

Applicable scenarios : Most e-commerce, payment, order, points async businesses; internet's first-choice solution .

3. TCC Flexible Transaction (High Concurrency, High Performance)

Core principle : Try (resource validation & reservation), Confirm (execute business), Cancel (rollback release resources) three-phase compensation.

Pros : Lock-free, high performance, adapts to high concurrency, strong eventual consistency.

Cons : Extremely high code intrusion, large development cost, requires hand-writing three-phase logic.

Applicable scenarios : Seckill, inventory deduction, high-concurrency fund transaction core scenes.

4. SAGA Long Transaction (Long-Chain Complex Transactions)

Core principle : Split long transaction into multiple local sub-transactions, each with compensation logic; on any node failure, execute all compensations in reverse to roll back data.

Pros : Adapts to long chains, complex flows, long-duration transactions.

Cons : Lock-free, intermediate state data exists, compensation logic complex.

Applicable scenarios : Order fulfillment, process approval, logistics long-chain businesses.

III. Eight High-Frequency Online Distributed Transaction Faults (Full Production Coverage)

Summarizing 99% of online distributed transaction anomalies; all data inconsistency problems are categorized into the following eight faults, each with symptoms, root causes, emergency stopgap, and root-cure solutions.

1. Transaction Suspension Fault (Most Concealed, Hardest to Troubleshoot)

Fault symptom : Business executes successfully but transaction state stalls; scheduled task retries infinitely; no errors, no log anomalies; data remains inconsistent long-term.

Core root cause : Network timeout, node jitter; transaction interrupted halfway, state not updated, resources not released, forming zombie transactions.

Root-cure solution : Add transaction timeout mechanism, state fallback updates, scheduled cleanup of suspended transactions, force-close abnormal transactions.

2. Message Redelivery Causing Duplicate Transaction Execution

Fault symptom : Duplicate points issuance, duplicate inventory deduction, duplicate order generation, duplicate ledger writes.

Core root cause : MQ retries, network retries, callback timeout retries; distributed transactions lack idempotency guarantees.

Root-cure solution : Mandatory idempotency for all distributed transactions , based on transaction ID or order number for unique validation; duplicate requests intercepted directly.

3. Half-Transaction Problem (Partial Success, Partial Failure)

Fault symptom : Service A executes and persists successfully, Service B fails and rolls back; both ends data inconsistent, cannot auto-repair.

Core root cause : Cross-service calls lack transaction linkage, no failure compensation, no global state control.

Root-cure solution : Unified transaction state machine; all sub-transactions linked; failure triggers global compensation rollback; prohibit single-node standalone success.

4. TCC Empty Rollback & Suspension Exceptions

Fault symptom : Cancel callback executes before Try; empty rollback error; subsequent Try succeeds but cannot roll back; data corrupted.

Core root cause : Network reordering, retry mechanisms break TCC three-phase sequence.

Root-cure solution : Add transaction state pre-check; intercept empty Cancel if Try not executed; record phase execution logs to guarantee sequence correctness.

5. Local Message Table Message Loss

Fault symptom : Business data persisted successfully, no message delivered; downstream service not executed; single-sided success.

Core root cause : Message delivery fails after local transaction commit; scheduled task fails to scan; message state not updated; program crash loses task.

Root-cure solution : Scheduled task fallback scanning, failed message retry mechanism, message log traceability, offline re-send capability.

6. Transaction Timeout Causing Data Inconsistency

Fault symptom : Long transaction times out; upstream commits, downstream interrupts; transaction not closed; permanent data deviation.

Core root cause : Interface timeout, network timeout, downstream service stalls; long transaction lacks timeout safeguard.

Root-cure solution : Split long transactions into short transactions; set maximum transaction timeout; timeout auto-triggers compensation rollback.

7. Concurrent Transaction Resource Contention Chaos

Fault symptom : Under high concurrency, multiple transactions compete for same resource; inventory over-deduction, amount chaos, state overwrites.

Core root cause : No distributed lock, no resource contention control; concurrent transactions operate same data in parallel.

Root-cure solution : Add distributed locks for hot resources, optimistic lock version control, concurrency rate limiting; avoid parallel tampering of core data.

8. Compensation Logic Failure, Transaction Cannot Roll Back

Fault symptom : Main business fails; compensation rollback logic errors or aborts; data cannot be reverted; dirty data formed.

Core root cause : Compensation logic lacks exception safeguards; extreme scenarios not self-tested; depends on unstable external interfaces.

Root-cure solution : Compensation logic prioritizes availability; independent exception safeguards; failures enter manual review queue; eliminate compensation failure.

IV. Distributed Transaction Selection Pitfall Guide (Production Absolute Rules)

The vast majority of transaction faults are essentially selection-scene mismatch ; summarizing production universal selection rules:

General async businesses (notifications, points, logs, statistics): Prefer local message table — simple, stable, zero-fault, easy maintenance.

High-concurrency core businesses (seckill, inventory, payment deduction): Use TCC transaction — high performance, non-blocking.

Long-chain process businesses (order fulfillment, multi-level approval): Use SAGA transaction — adapts to long-duration scenes.

Low-concurrency finance core (reconciliation, settlement): Use XA strong transaction — prioritize strong consistency.

Universal bottom line for all scenes : Prohibit cross-service data operations without transaction solution support; eliminate naked transactions.

V. Universal Online Fault Troubleshooting SOP for Distributed Transactions

When data inconsistency or transaction anomalies appear online, directly apply this closed-loop troubleshooting flow to quickly locate root cause:

Verify transaction state : Query transaction table, message table states; confirm undelivered, unconsumed, uncompensated, suspended timeout.

Reconstruct execution sequence : Replay Try/Confirm/Cancel, message produce/consume sequence from logs; investigate reordering, interruption issues.

Validate idempotency mechanism : Check for duplicate execution, missing idempotent interception causing data corruption.

Investigate network & timeouts : Verify interface timeouts, network jitter, node restarts causing transaction interruption.

Validate compensation logic : Confirm compensation/rollback logic executes normally after failure; check for errors.

Emergency data repair : Manual data reconciliation, offline re-send, manual closure of zombie transactions.

Long-term safeguard optimization : Complete idempotency, timeout, retry, monitoring, alerting mechanisms; prevent recurrence.

VI. Transaction High-Availability Safeguard Standards (Production Mandatory)

To thoroughly eliminate distributed transaction faults, all projects must enforce four safeguard standards:

Full-chain idempotency : All distributed transaction interfaces, message consumption, callback interfaces must implement idempotency.

Full-state observability : Monitor transaction initialization, executing, success, failure, timeout, suspension — all states; real-time alert on anomalies.

Full-exception compensability : Predefine compensation plans for all failure scenarios; no irreversible business operations.

Full-zombie cleanability : Scheduled cleanup of timeout, suspended, invalid zombie transactions; avoid resource accumulation, data backlog.

VII. Article Summary

As the first advanced bonus episode, this article thoroughly dissects distributed transaction underlying theory, four mainstream solutions, eight high-frequency faults, selection rules, and troubleshooting SOP , filling the most core and weakest data consistency capability in microservice architecture.

Thus, from basic troubleshooting, performance tuning, architecture safety nets to distributed data consistency, the entire system is fully perfected, completely covering junior developer → senior developer → architect full-stage production and interview needs.

VIII. Next Episode Preview

Next advanced bonus episode 2: Microservice Gateway & Registry Center Deep Fault Troubleshooting , dissecting Nacos/Eureka registration anomalies, service up/down jitter, heartbeat timeout, registration discovery inconsistency, gateway routing failure, and other high-frequency hidden faults, thoroughly solving all difficult problems in the microservice registration discovery layer.