TiDB HTAP Journey: From Transactional Roots to Hybrid Transactional/Analytical Processing

In this talk, PingCAP VP of Technology Shen Li shares the HTAP story of TiDB from the perspective of an AP (analytical processing) viewpoint, covering past explorations, current status, and future plans.

Initially TiDB was built as a TP (transaction processing) database to replace MySQL sharding and simplify user operations, but early users quickly demanded analytical capabilities as well.

The first low‑version focused on horizontal scaling: data is sharded across nodes, PD schedules the shards, TiKV stores the data, and TiDB provides a MySQL‑compatible front‑end. The architecture enables high availability and unlimited storage by adding nodes.

Early users appreciated that TiDB removed the need for manual sharding, but they were initially cautious about using it for critical transactional data. Some early workloads were log storage or simple analytics, and a parallel hash‑join implementation demonstrated better performance than MySQL.

As more users adopted TiDB, the system faced new challenges: large‑scale data synchronization, rule‑based data filtering, and increasing compute pressure on TiKV during AP workloads.

During the TiDB alpha stage, the distributed compute framework was rudimentary. In the 1.0 beta release, TiKV was extended to offload part of the computation: TiDB pushes operators to TiKV, performs filter, aggregation, and limit locally, then returns partial results for final aggregation on the TiDB server. Custom join operators were added to improve performance on large data sets.

Users found TiDB simplified data aggregation compared with manual ETL pipelines: no need to extract data to Hadoop/Hive, no complex Excel merging, and real‑time analytics became possible.

However, AP workloads exposed limitations: slow queries, OOM errors, CPU bottlenecks on TiKV, lack of automatic operator selection, and the need to still use MySQL protocol for external integration, requiring ETL or Spark/JDBC bridges.

To address the compute weakness, TiDB integrated TiSpark. TiSpark adds a TiDB driver to Apache Spark, allowing Spark’s optimizer to push simple scans, aggregations, and filters down to TiKV, while Spark handles heavier joins. This creates a hybrid MPP execution engine that leverages Spark’s ecosystem.

TiSpark has been adopted by users such as Yiguo Fresh, enabling near‑real‑time analytics by consolidating multiple data sources into a single TiDB cluster.

Beyond TiSpark, TiDB’s own optimizer has evolved from rule‑based (RBO) to cost‑based (CBO) with better statistics collection, batch execution, vectorized operators, and Apache Arrow‑based data exchange, reducing memory consumption and improving throughput.

Performance benchmarks (TPC‑H) show that TiDB 2.0 is significantly faster than 1.0, though some queries remain slow due to single‑threaded hash aggregation. TiDB 2.1 introduced multi‑threaded aggregation operators, further boosting performance.

Core contradictions remain: row‑store (TP) prefers low‑latency row format, while column‑store (AP) prefers columnar layout. TiFlash was created as a columnar extension of TiKV, synchronizing data via Raft logs and reusing ClickHouse code for vectorized execution.

TiFlash currently operates as a Raft learner, pulling row data from TiKV, converting it to columnar format, and serving queries with minimal impact on TiKV. Future versions will merge row and column stores, allowing the optimizer to treat columnar storage as a special index and enable MPP‑style joins across TiKV and TiFlash.

Current TiDB architecture already supports unlimited scaling, but challenges remain: full‑table scans on row‑store are inefficient, AP workloads can starve TP workloads, and there is no CPU pre‑emptive scheduling for mixed queries.

Future work includes building a native MPP engine for TiFlash, enabling data exchange between TiFlash nodes, pushing more computation down to storage, and improving large‑batch write paths to support multi‑gigabyte atomic writes.

Overall, TiDB aims to provide a unified platform where TP and AP workloads coexist without interfering, leveraging TiKV, TiFlash, TiSpark, and the TiDB Operator for Kubernetes‑based management.