go-mysql-transfer: A Go-Based Real-Time MySQL Binlog Incremental Sync Tool

Overview

The author evaluated Alibaba's open‑source Canal component and found three drawbacks: the need to write custom clients, the overhead of deploying both server and client for multiple databases, and extra network/serialization steps. go-mysql-transfer, written in Go, addresses these issues with a lightweight, one‑click deployable pipeline.

Features

No external dependencies; single‑binary deployment.

Built‑in support for Redis, MongoDB, Elasticsearch, RocketMQ, Kafka, RabbitMQ, eliminating client development.

Rich data parsing and message generation rules; Lua scripting for complex logic.

Prometheus exporter for monitoring and alerts.

High‑availability clustering via Zookeeper or etcd.

Failure retry and full‑data initialization capabilities.

Design and Implementation

1. Principle go-mysql-transfer masquerades as a MySQL slave, uses the dump protocol to fetch binlog, parses it, and streams messages to downstreams.

2. Data Conversion Rules Users define YAML‑style rules to map tables to target structures, e.g., syncing t_user to Redis strings with a key prefix.

rule:
  -
    schema: eseap # database name
    table: t_user # table name
    column_underscore_to_camel: true
    datetime_formatter: yyyy-MM-dd HH:mm:ss
    value_encoder: json
    redis_structure: string
    redis_key_prefix: USER_
    redis_key_column: USER_NAME

3. Lua Scripts Lua provides lightweight extensibility for custom data transformation. Example script loads JSON and Redis modules, builds a result object, encodes it, and issues SET or DEL commands.

local json = require("json")
local ops = require("redisOps")
local row = ops.rawRow()
local action = ops.rawAction()
local id = row["ID"]
local userName = row["USER_NAME"]
local key = "user_"..id
if action == "delete" then
    ops.DEL(key)
else
    local password = row["PASSWORD"]
    local createTime = row["CREATE_TIME"]
    local result = {id=id, userName=userName, password=password, createTime=createTime, source="binlog"}
    local val = json.encode(result)
    ops.SET(key, val)
end

4. Monitoring & Alerts The built‑in Prometheus exporter exposes metrics such as leader state, destination health, insert/update/delete counts, and replication delay.

enable_exporter: true
exporter_addr: 9595

5. High Availability Cluster mode (Zookeeper or etcd) ensures a single leader handles binlog dumping while followers stay idle; failover occurs within seconds.

cluster:
  name: myTransfer
  zk_addrs: 192.168.1.10:2181,192.168.1.11:2182,192.168.1.12:2183

6. Failure Retry go-mysql-transfer records the binlog position of failed batches and replays them after recovery, guaranteeing no data loss.

7. Full Data Initialization The -stock command performs a bulk SELECT‑based load, applies conversion rules or Lua scripts, and streams the initial dataset to the target.

Installation

Binary Download the pre‑compiled release from GitHub .

Source Build Requires Go 1.14+, set GO111MODULE=on, fetch the repo with go get -d github.com/wj596/go-mysql-transfer, then go build.

Deployment & Running

Enable MySQL binlog (ROW format) and set a unique server_id. Run the binary after editing app.yml, or launch via Docker after building the image.

# my.cnf
log-bin=mysql-bin
binlog-format=ROW
server_id=1

Performance Tests

Tests on a VM (E7‑4890, 8 GB RAM) using a 527 k row t_user table showed:

Full load to Redis via rules: ~4.6 s.

Full load via Lua: ~9.5 s.

Incremental sync (52 k rows) via rules: 32,950 TPS.

Incremental sync via Lua: 15,819 TPS.

100 concurrent writers: ~4,000 TPS (rules) and ~2,000 TPS (Lua) with stable resource usage.

Results may vary with environment.