How TiDB Achieves Multi‑Datacenter High Availability with Multi‑Raft and TiCDC

Introduction

TiDB is a distributed, financial‑grade, highly available database that stores data with multiple replicas synchronized via the Multi‑Raft protocol. Transactions are committed only after a majority of replicas acknowledge, guaranteeing strong consistency while tolerating failures of minority replicas. TiDB also allows configuring replica locations and counts to meet various disaster‑recovery requirements.

TiDB Multi‑Datacenter High Availability Overview

2.1 Single‑Cluster Same‑Domain Multi‑Datacenter

In this model a TiDB cluster is deployed across three data‑centers within the same region. Raft replicates data between the data‑centers, allowing any data‑center to serve read/write traffic. If one data‑center fails, the remaining two continue to provide consistent service.

Advantages:

All data replicas are distributed across three data‑centers, providing high availability and disaster recovery.

Zero data loss (RPO = 0) when any availability zone fails.

Automatic leader election and service recovery within ~20 seconds after a zone failure.

No manual intervention required; data consistency is maintained.

Disadvantages (network latency impact):

Write latency roughly doubles the inter‑zone latency because writes must be replicated to at least two zones using a two‑phase commit.

Read latency increases if the read‑request node is in a different zone from the leader.

Each transaction must obtain a TSO from the PD leader; cross‑zone TSO requests add latency.

2.2 DTS Multi‑Cluster Data Synchronization

A primary‑secondary TiDB cluster pair is built, and the TiCDC tool (used internally by HULK) synchronizes data from the primary to the secondary.

Advantages:

All replicas reside in the same data‑center, eliminating network‑induced latency.

The secondary cluster can be used for analytical workloads, reducing impact on the primary.

Limitations:

Does not support DDL/DML on system tables (e.g., mysql.*, information_schema.*) or temporary tables.

Does not support DQL and DCL statements.

Large table sync requires enabling enable-table-across-nodes = true (available from TiCDC v7.1.0).

Index operations ( ADD INDEX, CREATE INDEX) are executed asynchronously to reduce changefeed delay.

Large transactions (>5 GB) may cause increased latency, ErrBufferReachLimit errors, or OOM; enabling transaction splitting via transaction-atomicity mitigates this.

Failover requires updating the VIP or domain name, which may incur DNS propagation delay.

High‑Availability Principles

3.1 Single‑Cluster Same‑Domain Principle

TiDB uses label‑based placement. TiKV, a Multi‑Raft system, splits data into Regions (default 96 MiB). Each Region has three replicas forming a Raft group. Labels (e.g., zone, host) are reported to PD, which schedules replicas according to these labels to improve availability.

Key configuration parameters:

replication.location-labels: ["zone", "host"]
isolation-level: "zone"
max-replicas: 3
max-store-down-time: 30m

When isolation-level is set to zone, PD ensures that replicas of the same Region are placed in different zones. If a zone becomes unavailable, PD will not schedule new replicas to that zone, even if only two replicas remain, preserving the isolation constraint.

3.2 DTS Multi‑Cluster Synchronization Principle

TiCDC consists of multiple stateless nodes. Each node runs a Capture process; one Capture is elected as Owner, handling load scheduling, DDL sync, and management tasks. Captures contain Processor threads that pull table changes from the upstream TiDB cluster.

Data‑flow components:

Puller – fetches DDL and row changes from TiKV.

Sorter – orders changes by timestamp.

Mounter – converts changes to TiCDC‑compatible format.

Sink – applies changes to the downstream system.

High availability is achieved by running multiple TiCDC nodes that report status to PD’s etcd cluster and elect an Owner. If the Owner fails, another Capture becomes Owner, and processors are re‑scheduled to healthy nodes.

HULK Cloud Platform Usage

4.1 Single‑Cluster Same‑Domain Deployment

Public Serverless clusters have been upgraded to same‑domain multi‑datacenter HA. Monitoring shows each of the three zones (shrxx, shcxx, shyxx) holds 163 GiB, with roughly 9.6 K regions per zone, evenly distributed.

Query latency remains low; 95 % of queries complete within 6.29 ms, comparable to single‑zone deployment.

4.2 DTS Multi‑Cluster Deployment

TiCDC synchronizes primary‑secondary clusters for HA and analytical workloads. Monitoring dashboards display replication lag and alert status.

DBAs monitor delay and service‑alive alerts to ensure continuous availability.

Conclusion

Both the single‑cluster same‑domain and DTS multi‑cluster approaches provide strong HA for TiDB. The DTS architecture offers zero‑impact writes, cross‑region RPO in seconds, and RTO in minutes, while allowing the secondary cluster to serve analytical queries. The single‑cluster approach delivers zero RPO and minute‑level RTO when network latency is low. Users should choose the solution that fits their business requirements, and automation for HA deployment is under development.