Mongosync: Fast Full and Real-Time MongoDB Sync with Benchmarks

Background

Mongosync is a tool co‑developed by Qihoo 360’s Web Platform and DBA teams, built on the MongoDB protocol and a customized MongoDB sync module to enable data transfer between replica sets.

It is already used internally at 360, with source code and wiki available on GitHub: https://github.com/qihoo360/mongosync.

Features

Full data synchronization – copies data between MongoDB nodes, supporting cluster, database, collection, and query‑filtered modes. Example command for collection‑level sync without indexes:

./mongosync --src_srv 192.168.1.1:27017 --src_user user --src_passwd 123456 --src_auth_db admin --dst_srv 192.168.1.2:27018 --dst_user user --dst_passwd 123456 --dst_auth_db admin --db src_db --dst_db dst_db --coll src_coll --dst_coll dst_coll --bg_num 10 --no-index

Real‑time data synchronization – combines full sync with oplog streaming. Example command:

./mongosync --src_srv 192.168.1.1:27017 --src_user user --src_passwd 123456 --src_auth_db admin --dst_srv 192.168.1.2:27018 --dst_user user --dst_passwd 123456 --dst_auth_db admin --db src_db --dst_db dst_db --coll src_coll --dst_coll dst_coll --oplog

The --oplog flag continues real‑time replication after the initial bulk copy, and --op_start can be used for breakpoint resume.

Oplog real‑time transfer – streams the oplog between clusters, databases, or collections, with optional start and end timestamps.

Improvements and Innovations

Version compatibility – supports MongoDB 2.4, 2.6, 3.0 (partial support for 3.2/3.4).

Multithreaded bulk writes – --bg_num creates multiple parallel connections; effective when the target runs MongoDB 3.x.

Index filtering – --no-index skips index transfer to speed up data‑only sync.

Oplog‑only mode – transfers only the oplog with configurable timestamps.

Custom target database and collection names – allows renaming during sync.

Configuration‑file mode – can launch via a config file instead of command line.

Adjustable log levels – control verbosity of logs.

Performance Data

Full‑data sync test (≈10 GB, 13,588,980 records): 147 seconds, average 92 k QPS, network throughput ~67 MB/s. Ping latency: ~0.15 ms to source, ~1.23 ms to destination.

Oplog sync test (≈6.2 million records): 366 seconds, average 17 k QPS. Ping latency: ~10.8 ms both ways.

Future Plans

Extend compatibility to newer MongoDB versions (3.2, 3.4).

Support various source‑target topologies (single node ↔ replica set, mongos ↔ replica set, etc.) and real‑time oplog transfer under mongos.

Deeply optimize index creation during full sync to further boost speed.