Lightweight and Efficient Temporal Database Management System in TDSQL (T‑TDSQL) – VLDB 2019 Industry Track Paper

The VLDB conference is one of the three top-tier international conferences in data management and databases (SIGMOD, VLDB, ICDE). In 2019, Tencent and Renmin University of China jointly presented a paper titled “A Lightweight and Efficient Temporal Database Management System in TDSQL” that was accepted to the VLDB 2019 Industry Track and will appear as a full paper.

Since 2017, Tencent’s TEG Billing Platform team and the Key Laboratory of Data Engineering and Knowledge Engineering at Renmin University have collaborated on frontier database research. Their joint work has been selected for VLDB for two consecutive years.

The paper introduces a full‑temporal database built on top of Tencent’s distributed OLTP database TDSQL. The system retains the high OLTP performance of TDSQL while providing native temporal data management capabilities.

System Overview

The system adopts a hybrid storage model that separates current data and historical data. Current data use a page‑based MVCC structure with a dedicated rollback segment for old versions, which are later migrated to the historical store. Historical data are stored in a key‑value format, keeping only the attribute changes between successive versions.

For temporal queries, a dedicated compiler rewrites temporal predicates into standard WHERE clauses. Transaction‑time queries are transformed into visibility checks, ensuring transactional consistency. The system also supports time‑travel queries, fast historical version lookup via the KV store, and seamless use of standard SQL.

The architecture follows the HTAC (Hybrid Transaction/Analytical Cluster) design: an OLTP cluster handles transactional workloads, while an OLAP cluster processes analytical queries on historical data. A unified routing module directs SQL statements to the appropriate cluster, minimizing performance impact on production workloads.

Contributions

The paper proposes an extended temporal data model that adds a transaction‑ID attribute (MIN_ID, MAX_ID) to capture the creation and deletion of records, enabling precise reconstruction of data history.

An integrated temporal database solution is implemented in TDSQL, featuring asynchronous data migration, incremental historical storage, and a native temporal query executor, achieving a lightweight yet high‑performance system.

Extensive experiments on TPCC benchmarks and real‑world batch processing workloads show that the temporal extension incurs less than 10% performance loss compared to the original TDSQL, while significantly improving temporal query speed and reducing business execution time by nearly 50% in a 30‑day production test.

The VLDB reviewers summarized the work as an extension of Tencent’s TDSQL for temporal data management, presenting a new temporal data model, optimized implementation, and thorough experimental evaluation.

Significance

The full‑temporal capabilities provide accurate historical data retrieval, enhanced data safety, efficient indexing of temporal data, and support for data lineage and analytics, thereby offering a solid foundation for data security, replay, mining, and AI applications.