From CloudDBA to SDDP: Building Alibaba’s Self‑Driving Database Platform

Rooting (May 2016 – Aug 2019)

In mid‑2016, a senior leader assigned a small team to create a database diagnostic and optimization product called CloudDBA. The author, with four‑five teammates, began building the product despite limited initial understanding.

CloudDBA’s original goal was to translate DBA expertise into a self‑service diagnostic tool, addressing the rapid growth of Alibaba’s database footprint that could no longer be supported by manual DBA work. Early efforts produced a tool‑like prototype.

Recognizing the need for a product rather than a tool, the team defined a clear positioning statement: “CloudDBA is a database diagnostic‑optimization product for developers, aiming to become the database expert beside the user.” This shifted focus to productization, target users, and a vision slogan.

Product planning expanded functionality from single‑SQL diagnostics to comprehensive database diagnostics, leveraging the experience of a seasoned DBA team. The first internal version, codenamed “Sandwiches,” was launched after about six weeks of development.

During the next months, the team iterated bi‑weekly, bringing in a new product manager with DBA background, which deepened feature completeness.

Facing a performance bottleneck, the team realized that CloudDBA’s diagnostic capability was limited by sparse input data. They built a real‑time data pipeline that captured full‑SQL logs from the database kernel with less than 5 % overhead, streamed them via a message queue, and processed them with JStorm for the diagnostic engine.

By September 2016 the full‑SQL collection switch was rolled out across all clusters just before the Double‑11 shopping festival, a risky decision that proved successful: the pipeline handled peak loads of millions of SQL statements per second with only minute‑level latency spikes.

With abundant data, the team explored intelligent optimization. They identified two high‑impact use cases—anomaly detection and capacity prediction—and pursued deep‑learning methods, collaborating with Tsinghua’s Netman lab. This work resulted in two papers accepted at WWW 2018 and INFOCOM 2019.

Inspired by Andy Pavlo’s “Self‑driving Database Management Systems” (CIDR 2017), the author sought broader self‑driving capabilities. Although direct collaboration did not materialize, the paper guided the roadmap toward an end‑to‑end autonomous SQL optimization flow.

Public presentations at DTCC 2017, APMCon 2017, and Oracle OpenWorld 2017 highlighted CloudDBA’s progress and positioned it as China’s first intelligent database optimization product.

In 2018 the team launched automatic SQL optimization across the entire Alibaba fleet, marking a transition to a self‑driving database vision. They defined a “Database Self‑Driving Platform” that would accept SLA specifications as input and autonomously manage lifecycle, scaling, optimization, diagnosis, recovery, and security.

Organizationally, the CloudDBA team merged with another group to accelerate development, rebranding the product as SDDP (Self‑driving Database Platform). The unified platform established a closed‑loop “perception‑decision‑execution” architecture and incorporated collaborations such as automatic buffer‑pool tuning (paper “iBTune” published at VLDB 2019).

By 2019, the author presented the SDDP journey at the ICDE 2019 workshop, sharing lessons learned: define the right problems, build a flywheel, and iterate relentlessly to achieve breakthroughs.

The narrative concludes with reflections on the hardships, doubts, and triumphs experienced throughout the five‑year evolution from a modest tool to a full‑scale self‑driving database platform.