How JDTX Redefines Distributed Transactions with ACID Guarantees

In the era of distributed databases, cloud‑native databases, and NewSQL, transformation is inevitable. Incremental enhancements are preferred over radical redesigns in mature industries. Transparent data sharding, horizontal partitioning, elastic scaling, and HTAP are core concerns. JD.com’s open‑source Apache ShardingSphere has matured in data sharding, and the distributed transaction middleware JDTX builds on this foundation.

Background

Database transactions must satisfy the four ACID properties: atomicity, consistency, isolation, and durability.

In a single data node, transactions are local and are natively supported by most relational databases. In micro‑service based distributed applications, many scenarios require a single transaction to span multiple services and databases, giving rise to distributed transactions.

While relational databases provide perfect ACID support for local transactions, they become performance bottlenecks in distributed scenarios. Achieving ACID in a distributed setting or finding suitable alternatives is a key challenge.

Local Transaction

Without a distributed transaction manager, each node manages its own transaction independently, with no coordination or visibility of other nodes' transaction outcomes. Local transactions have no performance overhead but cannot guarantee strong or eventual consistency across nodes.

Two‑Phase Commit

The XA protocol, defined by the X/Open DTP model, is the earliest distributed transaction model. It offers minimal intrusion and transparent usage, strictly preserving ACID properties.

However, strict ACID enforcement requires locking all required resources for the transaction duration, making XA suitable only for short, low‑concurrency transactions. In high‑concurrency scenarios, XA’s performance degrades significantly.

Flexible Transactions

Rigid transactions satisfy ACID; flexible transactions follow the BASE model (Basically Available, Soft state, Eventual consistency). By moving lock handling from the resource layer to the business layer, flexible transactions relax strong consistency and isolation requirements, allowing higher throughput. Saga and TCC are typical implementations.

Conclusion

Both two‑phase commit (rigid) and flexible (BASE) transactions have trade‑offs. A comparison shows:

Two‑phase commit: no business changes required, full ACID support, but severe concurrency performance loss; suitable for short, low‑concurrency transactions.

Flexible transactions: require implementation of related interfaces, provide eventual consistency, slight performance degradation; suitable for long, high‑concurrency transactions.

There is no one‑size‑fits‑all solution for distributed transactions.

JDTX Distributed Transaction Solution

JDTX aims for strong consistency (full ACID), high performance (even surpassing local transactions), and a one‑phase commit (1PC) design, currently supporting relational databases. It uses an open SPI design, allowing future NoSQL integration and multi‑model transactions.

Design Philosophy

JDTX consists of a Transaction Manager (TM) and Resource Managers (RM).

The TM generates a globally monotonic transaction log sequence number (LSN), handles commit/rollback logic, and holds uncommitted tuples locally.

The RM manages active transaction data. Active data is written to a Write‑Ahead Log (WAL) and stored in an in‑memory MVCC engine, while persisted data is asynchronously flushed to the underlying relational database.

Queries merge persisted and active data according to the transaction’s isolation level to present a consistent view.

Key Highlights

Lossless Transaction Scheme

JDTX leverages WAL + MVCC to achieve true ACID semantics.

Atomicity & Consistency

The MVCC engine reduces two‑phase commit to a single phase, maintaining atomicity and consistency within a single node, effectively shrinking the distributed transaction scope to that of a local transaction.

Horizontal sharding and master‑slave sync enable scalability and high availability.

Isolation Support

JDTX implements snapshot isolation and repeatable read, covering most practical needs.

Durability Support

Active data is first persisted to WAL before entering the MVCC engine, ensuring recovery after crashes.

High Performance

Active data is asynchronously flushed to the database, shifting the bottleneck from DB writes to WAL and MVCC operations. The WAL follows sequential log appends, and the MVCC cache uses hash structures, making write latency lower than B‑Tree based databases. Consequently, JDTX can outperform transactions that keep the lock throughout.

Rollback is achieved without UNDO logs: uncommitted data never enters MVCC and is simply discarded locally.

High Availability

Both WAL and MVCC are sharded with master‑backup replication, eliminating single points of failure. If the MVCC engine becomes unavailable, data can be recovered from WAL to the underlying database.

Cross‑Database Transactions

Since active and persisted data are stored separately, the backend storage type is unrestricted. JDTX can maintain a single transaction semantics across heterogeneous stores such as MySQL, PostgreSQL, MongoDB, Redis, etc.

Implementation Challenges

MVCC Kernel

Most relational databases implement MVCC; JDTX adopts a PostgreSQL‑like approach, using xmin/xmax timestamps to track tuple visibility. Multi‑version chains are stored as linked lists, and visibility is determined by the transaction snapshot.

Currently only Snapshot Isolation (SI) is supported; Serializable Snapshot Isolation (SSI) is pending.

Vacuuming (clean‑up) of MVCC data is critical because long‑running transactions generate many versions, consuming memory. JDTX must decide when data has been flushed to WAL before reclaiming memory.

SQL Query Engine

Because the MVCC engine is not a relational DB, JDTX reuses ShardingSphere’s SQL parser and AST to map SQL queries to MVCC data.

For OLTP SPJ queries, primary keys are extracted, persisted data is fetched, and active data is merged. For OLAP aggregation queries, the engine rewrites SQL to fetch only aggregated results from the backend and merges them with active data.

Usage Limitations

All database access must go through JDTX; bypassing the middleware yields inconsistent transaction data.

SQL compatibility is still evolving; some dialect features may be missing.

JDTX requires primary keys for merging; tables without primary keys are unsupported.

JDTX and Apache ShardingSphere

Through ShardingSphere’s JDBC driver, JDTX can be seamlessly integrated into Java applications. ShardingSphere also offers MySQL and PostgreSQL proxy adapters, allowing JDTX to appear as a standalone database providing distributed transaction services.

ShardingSphere defines a unified SPI for distributed transactions; JDTX implements this SPI, enabling easy integration into the ShardingSphere ecosystem. Combined, they provide data sharding and distributed transaction capabilities.

Future Roadmap

JDTX aims to become a standard solution for distributed transactions. After stabilizing core components (transaction flow, MVCC engine, WAL, HA), the focus will be on:

Improving SQL compatibility and multi‑database support.

Implementing SSI isolation level for a complete MVCC isolation suite.

Enhancing management and monitoring tools.

JDTX will further integrate with ShardingSphere and cloud‑native platforms such as Kubernetes to serve cloud‑native databases.