The Evolution and Architecture of TiDB: From MySQL Compatibility to Distributed Storage

In 2015, after experience with MySQL sharding and middleware, PingCAP co‑founder and CTO Huang Dongxu decided to create a database that would eliminate the need for manual sharding, inspired by Google Spanner and F1 papers.

The team set a clear goal: a MySQL‑compatible, highly extensible database that minimized migration cost. They initially reused MySQL front‑end code and built a simple distributed storage engine, but performance was unacceptable, leading them to rewrite the SQL parser, optimizer, and execution layer in Go.

Early prototypes (version 0.2) lacked a real storage engine, so they abstracted storage behind thin interfaces, allowing any KV engine (LevelDB, RocksDB, etc.) to be plugged in. Extensive testing—over ten million test cases covering unit, logic, and ORM tests—became the cornerstone of correctness.

Following the "Talk is cheap, show me the tests" principle, TiDB enforced strict test coverage before any code could be merged, emphasizing reproducible failure scenarios and full system replay using captured TCP/IP packets.

As the SQL layer matured, the architecture progressed to version 0.5, decoupling SQL from storage and integrating a distributed engine (initially HBase, later replaced). The storage layer ultimately settled on RocksDB, a high‑performance LSM‑tree key‑value engine.

To implement a reliable consensus algorithm, the team translated Etcd’s Raft implementation line‑by‑line into Rust, ensuring identical test behavior. Rust was chosen for its performance, safety, and lack of garbage collection, becoming the language for TiKV, the storage component.

TiKV, open‑sourced in April 2016, combined RocksDB for local storage and Raft for replication, delivering elastic scaling, ACID transactions, global consistency, and multi‑data‑center high availability.

Metrics and observability were addressed by embedding Prometheus (a distributed time‑series database) and Grafana for dashboards, enabling fine‑grained performance monitoring and alerting.

The article also stresses engineering practices: minimizing migration friction, ensuring MySQL‑compatible behavior, anticipating failures, and building cloud‑native, self‑healing systems that can automatically rebalance and recover.

Overall, TiDB’s story illustrates a disciplined approach to building foundational software—starting from a clear compatibility goal, iterating through layered abstractions, rigorous testing, and leveraging open‑source components to achieve a production‑grade distributed database.