How JDTX Achieves High‑Performance Distributed Transactions with MVCC and WAL

Background

Modern database ecosystems contain many concepts such as distributed databases, cloud‑native databases, and NewSQL. Transparent data sharding is the core technique for handling massive data, while elastic scaling and HTAP are also gaining attention. JD.com’s open‑source Apache ShardingSphere provides mature data‑sharding capabilities, and JDTX builds on it to form a distributed transaction kernel.

Transaction Fundamentals

Local Transactions

Each node manages its own transaction independently, without coordination or visibility across nodes. This yields excellent performance but weak consistency.

Two‑Phase Commit (2PC)

The XA protocol implements 2PC with minimal business intrusion and strict ACID guarantees. All required resources are locked for the whole transaction, making it unsuitable for long‑running, high‑concurrency workloads because performance degrades sharply.

Flexible (BASE) Transactions

BASE (Basically Available, Soft state, Eventual consistency) relaxes consistency and isolation, moving lock handling to the business layer. Saga and TCC are typical implementations that trade strict ACID for higher throughput.

JDTX Architecture

JDTX aims for strong consistency (full ACID), high performance (even surpassing local transactions), and a one‑phase commit (1PC) model. It consists of a Transaction Manager (TM) and a Resource Manager (RM) and exposes an open SPI for future NoSQL integration.

Key Components

Transaction Manager : Generates globally monotonic transaction log sequence numbers (LSN), handles commit/rollback, and holds local tuples of uncommitted transactions.

Resource Manager : Separates active (in‑transaction) data from persisted data. Active data is first written to a Write‑Ahead Log (WAL) and then stored in a custom MVCC in‑memory engine; persisted data is asynchronously flushed to the underlying relational database.

Transaction Processing Flow

Queries within a transaction merge persisted data with active MVCC data according to the transaction’s isolation level, providing a consistent view. The MVCC engine currently supports read‑committed and repeatable‑read isolation levels.

Solution Highlights

Lossless Transaction Model

By combining WAL and MVCC, JDTX preserves the original ACID semantics without the overhead of traditional 2PC.

Performance

Active data is asynchronously flushed, shifting the bottleneck from database writes to WAL and MVCC operations. The WAL follows an append‑only log, and the MVCC cache uses hash‑based structures, making write latency lower than B‑Tree‑based databases. JDTX also eliminates UNDO logs; uncommitted data remains in the TM and is discarded on rollback, further boosting performance.

High Availability

Both WAL and MVCC engines are sharded and replicated, avoiding single points of failure. If the MVCC engine becomes unavailable, data can be recovered from WAL to the database.

Cross‑Database Transactions

Because active and persisted data are stored separately, JDTX can span heterogeneous back‑ends (MySQL, PostgreSQL, MongoDB, Redis, etc.) while maintaining a single transaction semantic.

Implementation Challenges

MVCC Kernel

JDTX adopts a PostgreSQL‑like MVCC scheme, tagging each tuple with xmin / xmax transaction IDs. Multi‑version chains are stored as linked lists, and visibility is determined by comparing transaction snapshots.

Isolation Levels

Current support includes Snapshot Isolation (SI), which maps to read‑committed and repeatable‑read in ANSI. Full Serializable Snapshot Isolation (SSI) is not yet implemented, and MySQL’s lack of SSI means JDTX presently offers only SI.

MVCC Garbage Collection

Long‑running transactions generate many MVCC versions, consuming memory. JDTX must vacuum obsolete versions and consider whether data has been flushed to WAL before reclamation.

SQL Query Engine

Since the MVCC engine is not a relational DB, JDTX leverages ShardingSphere’s SQL parser and AST to translate queries. For OLTP (select‑project‑join) queries, primary keys are used to merge active and persisted data. For OLAP aggregations, JDTX rewrites the SQL to query only persisted data and then merges results.

Usage Limitations

All database access must go through JDTX; bypassing the middleware yields inconsistent transaction data.

SQL compatibility is still evolving; reduced dialect support is a trade‑off for lossless ACID.

Operations on tables without primary keys are unsupported because key‑based merging is required.

Integration with Apache ShardingSphere

ShardingSphere provides a JDBC driver and proxy endpoints (MySQL, PostgreSQL) that allow Java applications to use JDTX transparently. The SPI in ShardingSphere lets JDTX plug into the ecosystem, enabling seamless combination of data sharding and distributed transactions.

Future Roadmap

Improve SQL compatibility and add support for more heterogeneous databases.

Implement SSI isolation level to provide a complete MVCC isolation suite.

Enhance management and monitoring interfaces.