How X‑Engine Redefines LSM Storage for High‑Performance E‑Commerce

Background

X‑Engine is Alibaba Cloud's self‑developed OLTP storage engine, used as the storage layer for POLARDB X and deployed across many internal business systems such as transaction history and DingTalk, dramatically reducing costs and handling hundreds of times the normal traffic during major sales events.

Architecture

Built on an LSM foundation, X‑Engine targets a balanced read‑write performance by heavily optimizing write paths, compaction impact, compression benefits, tiered hot‑cold storage, and fine‑grained caching.

The overall architecture is illustrated below:

LSM Basics

Data enters the system via a Write‑Ahead Log (WAL) and a MemTable. When the MemTable fills, it is flushed to a persistent level, and periodic compaction merges adjacent levels, discarding obsolete versions and shaping the LSM hierarchy.

Compaction is resource‑intensive because it must read, sort, and rewrite data across levels.

Refined LSM

X‑Engine uses a lock‑free SkipList for its MemTable and introduces several refinements to the classic LSM design, including tiered hot‑cold data placement, selective compaction scheduling, and extensive cache management.

Data Organization

Each LSM level is divided into fixed‑size Extents, which are further split into Data Blocks. Extents are indexed by a Meta Index, and together with active/immutable MemTables they form a Metadata Tree similar to a B‑Tree. All structures except the active MemTable are immutable; snapshots provide a consistent view for SI isolation.

Compaction creates new Extents and updates only the affected Meta Index nodes, achieving high data‑reuse rates.

Transaction Processing

Transactions are split into a read‑write phase (conflict detection, buffering changes) and a commit phase (WAL write, MemTable write, and client response). X‑Engine employs a four‑stage pipeline—Log Buffer, Log Flush, Write MemTable, Commit—allowing each stage to run in parallel across many threads, dramatically increasing throughput.

Read Operations

Single‑record lookups benefit from a Row Cache placed above all persistent layers; if a record is not found in the MemTable, the Row Cache can still serve the latest version. Range scans use additional techniques such as Surf filters, asynchronous I/O, and prefetching to reduce the cost of scanning multiple levels.

Compaction

Compaction merges adjacent levels by reading overlapping key ranges, discarding obsolete versions, and writing the result to new Extents. X‑Engine’s Extent‑based design enables fine‑grained compaction and extensive data reuse, reducing I/O, CPU usage, and write amplification while keeping cache invalidation minimal.

Conclusion

X‑Engine is a core technology of Alibaba Cloud’s database portfolio and will be offered as a MySQL‑compatible storage engine for POLARDB X on the public cloud, delivering low‑cost, high‑performance database services to a broad range of customers.