Mastering TiDB Binlog: Architecture, Scaling, and Data Recovery with Reparo

TiDB Binlog Introduction

TiDB Binlog collects TiDB database logs and provides data synchronization and near‑real‑time backup capabilities.

TiDB Binlog Architecture and Principles

TiDB Binlog consists of Pump and Drainer components. Pump nodes receive binlog events from TiDB instances, order them by transaction commit time, and forward them to Drainer. Drainer merges binlogs from all Pumps and converts them to a target format such as Kafka, another TiDB cluster, MySQL, MariaDB, or plain files.

Scaling Drainer Nodes

To expand a Drainer cluster, edit the drainer configuration (set syncer.db-type="file" to write binlog files), ensure binlog.enable and binlog.ignore-error are enabled in TiDB, and run:

tiup cluster scale-out <cluster-name> scale_out_drainer.yaml --user tidb -p

Verify the cluster status with: tiup cluster display <cluster-name> Check that binlog files exist in the configured directory, e.g.:

cd /data1/tidb-data/drainer-8249/binlog
ll -h

Parsing Binlog with Reparo

Reparo is a companion tool that parses TiDB binlog files (stored as protobuf) and can replay them. Install Reparo on the drainer host, then configure reparo.toml (set data-dir, dest-type="print" or "mysql", and optional time range). Example configuration snippets:

# Install Reparo
su - tidb
wget https://download.pingcap.org/tidb-binlog-cluster-latest-linux-amd64.tar.gz
tar -xvf tidb-binlog-cluster-latest-linux-amd64.tar.gz
mv tidb-binlog-cluster-latest-linux-amd64 tidb-binlog-cluster
cd /home/tidb/tidb-binlog-cluster/bin

# reparo.toml
[data-dir]
"/data1/tidb-data/drainer-8249/binlog"
log-level = "info"
start-datetime = "2022-07-27 16:42:00"
stop-datetime = "2022-07-27 16:51:00"
dest-type = "print"

Run Reparo:

reparo -config /home/tidb/reparo.toml > /home/tidb/binlog_parse.log

The tool outputs parsed DDL/DML statements and logs success messages such as:

[2022/07/27 16:53:55.748 +08:00] [INFO] [config.go:153] ["Parsed start TSO"] [ts=434873649070080000]
[2022/07/27 16:53:55.749 +08:00] [INFO] [config.go:160] ["Parsed stop TSO"] [ts=434873774899200000]

Experiment: Full Backup + Binlog Data Recovery

The experiment demonstrates restoring data after accidental deletion by combining a full BR backup with TiDB binlog.

Steps:

Obtain the backup TSO (e.g., BackupTS=433311088594911234) using br validate decode.

Restore the target table in a standby environment:

br restore table --pd "10.xxx.xxx.104:2379" --db "wangn_test" --table "tt" \
  --ratelimit 128 --storage "local:///bak/data1/wangn/tidb_test/7120519" \
  --log-file /tmp/restorefull.log

Determine the start and stop TSO from binlog logs (e.g., start‑tso = 433311088594911235, stop‑tso = 433314381739589634).

Configure a source Reparo instance to read the binlog files and replay them to the test MySQL target:

# reparo_source.toml
[data-dir]
"/bak/data1/wangn/binlog"
log-level = "info"
start-tso = 433311088594911235
stop-tso = 433314381739589634
dest-type = "mysql"
[dest-db]
host = "10.xxx.xxx.104"
port = 4000
user = "******"
password = "******"

reparo -config /home/tidb/reparo_source.toml > /home/tidb/binlog_parse_source.log

After execution, the data is restored in the test environment and can be exported back to production.

References: PingCAP documentation for TiDB Binlog overview and Reparo.