How Tencent's TDSQL‑C Won SIGMOD: Inside the Spitfire Three‑Tier Buffer Manager

From June 20‑25, 2021 the SIGMOD conference returned to China, held in Xi’an. ACM SIGMOD is the premier international conference in the database field.

Tencent Cloud’s latest research results were accepted as SIGMOD papers, highlighting the technical innovation of its cloud‑native database TDSQL‑C (formerly CynosDB).

Keynote by Qiu Min

On June 23, Tencent Cloud Database Director Qiu Min delivered a keynote titled “The Evolution of Tencent Database Technologies”, outlining the evolution of Tencent’s database products and focusing on the challenges and innovations of TDSQL‑C.

TDSQL‑C Overview

TDSQL‑C is a distributed, compute‑storage‑separated architecture that is 100 % compatible with MySQL and PostgreSQL. Its instance count has grown over 3400 % year‑over‑year, serving gaming, e‑commerce, live‑streaming, finance and SaaS workloads. The system achieves major gains in latency, scaling, HA and RTO, and supports up to 1 PB storage per instance and 1 M QPS, with a serverless mode.

During the recent 618 shopping festival, TDSQL‑C handled billions of QPS and petabytes of storage, ensuring smooth consumer experience.

SIGMOD‑Selected Paper: Spitfire

The paper “Spitfire: A Three‑Tier Buffer Manager for Volatile and Non‑Volatile Memory” introduces a low‑cost, effective hot‑data identification method that boosts overall system performance and cost‑effectiveness. It proposes a three‑tier buffer manager that keeps hot pages in DRAM, warm pages in NVM, and uses simulated annealing to find the optimal data‑migration strategy.

Implemented in TDSQL‑C, Spitfire improves read performance by over 100 % without significant cost increase, and received high praise from SIGMOD reviewers for its novel integration of NVM hardware with database architecture.

Hardware‑Software Co‑Design

Soft‑hardware co‑design is a core technology of TDSQL‑C, developed together with the Starry Sea Lab, leveraging new storage media, network and compute advances to deliver leading cloud‑native database services.

The architecture diagram above illustrates the multi‑tier buffer hierarchy and the separation of compute and storage in TDSQL‑C.