Design of High‑Availability System and Fast Recovery in Tencent CynosDB

On March 16, Tencent Cloud + Community held a CynosDB technical exchange in Beijing, providing a comprehensive overview of CynosDB, including its compatibility with two major open‑source databases, a one‑master‑multiple‑read architecture, high‑availability design, fast recovery, intelligent compute‑storage, and distributed storage.

Why high availability matters : Services must run 24/7, environments are unreliable (software bugs, hardware failures, natural factors), and data loss can cause severe business impact. Therefore, ensuring database availability is critical.

Industry HA solutions : Two main approaches are used in the industry. The first relies on consensus protocols (Paxos, Raft) such as MySQL 5.7 MGR, which provides strong consistency but requires at least three nodes, version‑specific support, and can be intrusive. The second uses an external monitoring system (e.g., MHA) that probes MySQL nodes, is version‑agnostic, and non‑intrusive, but adds operational overhead and its own HA requirements.

CynosDB adopts the external‑system approach, with a dedicated high‑availability (HA) service that itself is highly available.

Design decomposition : The HA design is split into three sub‑problems – fault detection, decision making, and recovery.

Fault detection methods :

Internal heartbeat (MySQL MGR) – requires ≥3 nodes.

External server probing (e.g., MHA Manager) – may miss local failures and can cause split‑brain.

Local agent deployed on the MySQL host – provides timely detection of process state, replication lag, and host load, and is the method chosen by CynosDB.

The agent monitors MySQL process status, replication threads, data lag, and host metrics. Typical RTOs are ≤5 s for process restart, ≤10 s for switch, and ≤30 s for full recovery.

Decision making : After a fault is detected, CynosDB classifies actions into three strategies – automatic switch for master failures, re‑join for slave process recovery, and rebuild when re‑join fails. The Scheduler issues HA tasks based on these strategies.

Fast recovery via VDL : CynosDB stores data in a distributed storage layer (TXStore). Recovery is driven by a “volume‑persistent sequence number” (VDL). Mini‑transactions (mtr) consist of multiple redo logs. The client broadcasts the last known VDL (L‑VDL) to all tablets, collects their completed persistent points (CPL), builds a min‑heap, and determines the highest continuous CPL as the new VDL. This process is illustrated with binary‑heap sorting and truncation of non‑persistent logs.

Overall HA architecture : Consists of an Agent (co‑located with mysqld) that continuously reports status, and a Scheduler (highly available via ZooKeeper) that orchestrates HA tasks such as switch, rejoin, and rebuild. The Agent is stateless and is restarted by Supervisor; the Scheduler ensures only one active instance at a time.

Handling split‑brain (brain‑split) scenarios : Each instance version number is stored in TXStore. When a network partition triggers a false master switch, the version number is incremented, causing the stale node’s writes to fail with a version‑mismatch error, forcing it to exit and preventing dual‑write situations.

The presentation concludes with a Q&A covering cross‑region consistency, data volume handling, migration benefits, performance improvements (reduced IO, lock splitting, thread‑pool optimizations), and the distinction between network IO and disk IO in CynosDB.