How to Build and Manage a Large‑Scale MongoDB Replica Set with Incremental Backups

Overview

A client implementation was completed in mid‑September, describing a production‑grade MongoDB replica‑set environment and step‑by‑step procedures for setup, configuration, incremental backup, and recovery.

1. Environment Construction

Online topology : a replica set with three business nodes, one hidden node (used for Hadoop/Spark sync and reporting), and one delayed node.

Hosts and priorities :

10.21.18.21 primary – priority 100

10.21.18.22 secondary – priority 90

10.21.18.23 secondary – priority 80

10.21.18.24 hidden – priority 0

System specs : 128 GB RAM, 64 CPU cores, 2 TB SSD.

MongoDB version : percona-server-mongodb-3.0.12-1.8 Data volume : total 1.3 TB; single collection up to 670 million documents; 11 collections > 500 million, 27 collections > 200 million. No sharding is used.

Configuration file (mguserinfo.conf) :

# systemLog
destination: file
logAppend: true
logRotate: reopen
path: /data/users/mgousr01/mongodb/logs/userinfo01.log

# storage
dbPath: /data/users/mgousr01/mongodb/dbdata
journal:
  enabled: true
directoryPerDB: true
engine: wiredTiger
wiredTiger:
  engineConfig:
    cacheSizeGB: 64
    directoryForIndexes: true
  indexConfig:
    prefixCompression: true

# operationProfiling
slowOpThresholdMs: 50
mode: "all"

# processManagement
fork: true
pidFilePath: /data/users/mgousr01/mongodb/dbdata/userinfo01.pid

# net
port: 28010
maxIncomingConnections: 27600

# security
keyFile: /data/users/mgousr01/mongodb/etc/keyFile/keyFilers0.key

# replication
replSetName: userRS
oplogSizeMB: 95360
secondaryIndexPrefetch: all

Note : keyFilers0.key is the authentication key file and must have permission 600.

2. Startup

Start the instance with:

mongod -f mguserinfo.conf

3. Replica‑Set Initialization

On the primary node:

config={_id:"userRS",members:[{_id:1,host:"10.21.18.21:28010",priority:100,tags:{'use':'usersinfo-01-312'}},{_id:2,host:"10.21.18.22:28010",priority:90,tags:{'use':'usersinfo-02-312'}},{_id:3,host:"10.21.18.23:28010",priority:80,tags:{'use':'usersinfo-03-312'}},{_id:5,host:"10.21.18.24:28010",priority:0,hidden:true,tags:{'use':'usersinfo-03-312-hidden'}}]}
rs.initiate(config)
rs.add({_id:2,host:"10.21.18.22:28010",priority:90,tags:{'use':'usersinfo-02-312'}})
rs.add({_id:3,host:"10.21.18.23:28010",priority:80,tags:{'use':'usersinfo-03-312'}})
rs.add({_id:5,host:"10.21.18.24:28010",priority:0,hidden:true,tags:{'use':'usersinfo-03-312-hidden'}})

Create an admin user with full privileges:

use admin
db.createUser({user:"admin",pwd:"123456",roles:[{role:"userAdminAnyDatabase",db:"admin"},{role:"readAnyDatabase",db:"admin"},{role:"dbOwner",db:"admin"},{role:"userAdmin",db:"admin"},{role:"root",db:"admin"},{role:"clusterMonitor",db:"admin"},{role:"dbAdmin",db:"admin"}]})

4. Incremental Backup Strategy

The process involves temporarily removing a secondary node, adjusting its authentication settings, performing an oplog‑based backup, and then reintegrating the node.

Shut down a secondary (e.g., 10.21.18.22) using mongod -f mguserinfo.conf --shutdown and remove it from the replica set with rs.remove("10.21.18.22:28010").

Modify mguserinfo.conf on that node to comment out replSetName, keyFile and security, then restart it in standalone mode.

After backup, restore the original parameters, restart, and add the node back with rs.add(...).

Switch the primary to the former secondary by lowering the original primary’s priority to 85 and reconfiguring:

cfg = rs.conf()
cfg.members[1].priority = 85
rs.reconfig(cfg)

Create a dedicated backup user on the new primary:

use admin
db.createUser({user:"bakinc",pwd:"Uk#R$K25A9",roles:[{role:"userAdminAnyDatabase",db:"admin"},{role:"readAnyDatabase",db:"admin"},{role:"dbOwner",db:"admin"},{role:"userAdmin",db:"admin"},{role:"root",db:"admin"},{role:"dbAdmin",db:"admin"}]})

5. Incremental Backup Script (mongo_inc.sh)

#!/bin/bash
DATA1=`date +%Y%m%d%H%M%S`   # current timestamp for backup directory name
DATA2=`date +%s`            # epoch seconds
DATA3=`echo $((${DATA2}-10860))`  # three hours and one minute earlier

echo $DATA2
echo $DATA3

/data/mongodb/users/mgousr02/mongobackup -h10.21.18.22 --port 28010 -u bakinc -p "Uk#R$K25A9" --authenticationDatabase admin \
  --backup -s ${DATA3},0 -t ${DATA2},0 -o backup${DATA1}

The script generates a new backup directory each run and can be scheduled to run every few hours.

6. Incremental Restore

Copy the backup directory to the target host and run:

./mongobackup -h10.66.168.68 --port 28010 --recovery -s 1475031191,0 -t 1475032271,0 ./backup20160928110254

Here -s is the start timestamp and -t the end timestamp.

7. Troubleshooting Authentication Errors

If the system.version collection is not updated, backups may fail with:

Failed to authenticate admin@admin with mechanism MONGODB-CR: AuthenticationFailed MONGODB-CR credentials missing in the user document

Solution: delete existing users (or keep them), set

db.system.version.update({_id:"authSchema"},{$set:{"currentVersion":3}})

, then recreate the required users.

8. Test Data Generation

Sample scripts to insert large volumes of documents for performance testing:

for(var i=1;i<=100000;i++) db.users.insert({_id:i,ip:"192.168.168.254",g_roup:"kiwi",mac:"of:fd:67:8c:2f:8f",address:"hongmei1801num",user_id:i,name:"user10000000"+i,title:"system",database:"mongodb",telphone:NumberLong("15718441234"),mail:"[email protected]",os:"win7",company:"chinapnr",created:new Date(),age:Math.floor(Math.random()*120)})
for(var i=1;i<=1000000;i++) db.infos.insert({...})
for(var i=1;i<=600000;i++) db.addrs.insert({...})

After a full restore, additional inserts/updates/deletes can be performed, followed by incremental backup and restore to verify correctness.

9. Backup Method Discussion

Three common approaches for MongoDB incremental migration:

Custom scripts that capture DML changes via the oplog, enabling point‑in‑time recovery.

Existing tools such as mongobackup that operate on the oplog.

Programmatic migration (e.g., Java) that moves historical data to another store like HBase.

All rely on the oplog; open‑source solutions are still maturing compared with the commercial MongoDB offering.