How TDDL Revolutionized Database Scaling at Alibaba: Lessons from 2007

Introduction

TDDL (Taobao Distributed Data Layer) is an open‑source middleware developed by Taobao to provide sharding, master‑slave, read‑write separation, weight allocation and dynamic configuration for MySQL databases.

2007 Context and Problems

At the end of 2007 Taobao had 400 billion CNY turnover and 10 million daily active users. The monolithic database suffered from:

20:1 read‑write ratio.

Single‑table size exceeding tens of millions of rows, causing seconds‑long queries.

All tables and databases hosted on a single machine, leading to disk and CPU bottlenecks.

Design Goals

Provide fast response for 10 million daily users and a scalable path to hundreds of millions.

General Solutions

SQL optimization (syntax cleaning, push‑down, index tuning).

Index creation on hot columns.

Vertical and horizontal sharding.

Read‑write separation.

Caching.

SQL Optimizer and Parser

Build a parser that converts a SQL statement into an abstract syntax tree, then apply rules such as constant folding, predicate push‑down and range merging. Example:

SELECT id, member_id FROM wp_image WHERE member_id = '123'

Sharding Strategies

Vertical sharding splits a table into a core table (frequently accessed columns) and an extension table (rare or large columns). Horizontal sharding distributes rows across multiple tables or databases based on a key.

Horizontal sharding is the core of TDDL. It requires:

Routing algorithm (fixed hash or consistent hash) to locate the target shard.

Result merging after parallel execution.

Globally unique primary keys.

(Future) distributed transaction support.

Routing Algorithms

Fixed hash uses modulo; consistent hash adds virtual nodes for better balance and easier scaling.

Global Unique ID Generation

Allocate a block of IDs (e.g., 100) from a database counter, then dispense them in memory, ensuring uniqueness across shards.

Distributed Architecture

Combine sharding, read‑write separation, backup, and dynamic expansion. Diagrams illustrate the data flow from client → TDDL → parser → optimizer → routing → physical DB.

Operational Practices

Prefer caching to reduce read load.

Apply read‑write separation before sharding.

Vertical then horizontal splitting for clear business boundaries.

Use weight‑based routing for gradual capacity expansion.

Conclusion

TDDL demonstrates how a middleware layer can evolve from a single‑machine MySQL deployment to a fully distributed, horizontally sharded system capable of supporting massive traffic.