Master MongoDB Replication Sets and Sharding: A Complete Step‑by‑Step Guide

1.1 MongoDB Replica Set Overview

A MongoDB replica set is a group of mongod processes that maintain the same data set, providing redundancy and high reliability for production deployments.

1.1.1 Purpose of Replica Set

Ensures data redundancy and reliability by storing copies on different machines, preventing data loss from single‑point failures, and improves read capacity by distributing read operations across members.

1.1.2 Simple Introduction

A replica set consists of multiple mongod instances sharing a data set. One primary handles all writes; secondaries replicate from the primary. If the primary fails, an election selects a new primary.

Each set also includes an arbiter that participates in elections without storing data.

1.2 Basic Architecture of Replication

A three‑member replica set includes three data‑bearing members or two data members plus an arbiter.

1.2.1 Three Data‑Bearing Members

One primary; two secondaries that can become primary if the primary fails.

When the primary fails, one secondary is elected as the new primary; the original primary rejoins as a secondary.

1.2.2 Arbiter Node

An arbiter only votes in elections and cannot become primary.

Arbiter nodes provide a vote without storing data, reducing resource usage but offering limited redundancy.

If the primary fails, the arbiter helps elect a new primary.

1.2.3 Primary Election

Replica sets are initialized with replSetInitiate or rs.initiate(). Members exchange heartbeats and elect a primary that receives a majority of votes.

Definition of Majority

If the number of voting members is N, a majority is N/2 + 1. If fewer than a majority are alive, the set cannot elect a primary and becomes read‑only.

It is recommended to use an odd number of voting members to tolerate failures.

1.3 Member Types in a Replica Set

1.3.1 All Members

Secondaries participate in elections, can become primary, and replicate data from the primary, providing read capacity and high availability.

Arbiter nodes only vote, never become primary, and do not sync data.

Priority 0 nodes have election priority 0 and will not become primary.

Hidden nodes cannot become primary and are invisible to drivers; useful for backups or reporting.

Delayed nodes are hidden nodes whose data lags behind the primary by a configurable delay, useful for recovery.

1.3.2 Priority 0 Nodes

Used as standby members, ensuring the primary resides in a specific data center.

1.3.3 Hidden Nodes

Clients do not read from hidden nodes; they can be used for reporting or backup tasks.

1.3.4 Delayed Nodes

Delayed nodes store data with a time lag, allowing recovery from accidental writes.

1.4 Configuring a MongoDB Replica Set

1.4.1 Environment

System environment: CentOS 6.9, kernel 2.6.32, firewall disabled, SELinux disabled, IP addresses 10.0.0.152 and 172.16.1.152.

1.4.2 Create mongod User

useradd -u800 mongod
echo 123456|passwd --stdin mongod

1.4.3 Create Directories

for i in 28017 28018 28019 28020; do
  mkdir -p /mongodb/$i/conf
  mkdir -p /mongodb/$i/data
  mkdir -p /mongodb/$i/log
done

1.4.4 Configure Multi‑Instance

Edit the first instance configuration file:

cat >>/mongodb/28017/conf/mongod.conf<<'EOF'
systemLog:
  destination: file
  path: /mongodb/28017/log/mongodb.log
  logAppend: true
storage:
  journal:
    enabled: true
  dbPath: /mongodb/28017/data
  directoryPerDB: true
  wiredTiger:
    engineConfig:
      cacheSizeGB: 1
      directoryForIndexes: true
    collectionConfig:
      blockCompressor: zlib
    indexConfig:
      prefixCompression: true
processManagement:
  fork: true
net:
  port: 28017
replication:
  oplogSizeMB: 2048
  replSetName: my_repl
EOF

Copy the configuration to the other instances and replace the port numbers.

1.4.5 Start Services

for i in 28017 28018 28019 28020; do
  mongod -f /mongodb/$i/conf/mongod.conf
done

1.4.6 Initialize Replica Set

Connect to the primary and run:

config = {_id: 'my_repl', members: [
  {_id: 0, host: '10.0.0.152:28017'},
  {_id: 1, host: '10.0.0.152:28018'},
  {_id: 2, host: '10.0.0.152:28019'}
]};
rs.initiate(config);

Replica set configuration is complete.

1.4.7 Test Primary‑Secondary Replication

Insert documents on the primary and verify they appear on secondaries using rs.slaveOk() and db.movies.find().

1.4.8 Replica Set Management

Check status with rs.status(), add or remove members with rs.add(), rs.addArb(), rs.remove(), and configure delayed or hidden members by modifying the replica set configuration and reconfiguring with rs.reconfig().

2 MongoDB Sharding

Sharding distributes large collections across multiple servers, providing horizontal scalability and high throughput.

2.1 Sharding Overview

2.1.1 Purpose

Horizontal scaling addresses performance bottlenecks of large data volumes and high query loads.

2.1.2 Design Philosophy

Sharding reduces per‑node load by routing operations to the appropriate shard.

2.1.3 Advantages

Abstracts the cluster, making it invisible to applications.

Ensures the cluster remains writable and readable.

Facilitates easy scaling.

2.1.4 Sharding Architecture

Components: Config servers store metadata, mongos routers handle client requests, and mongod shards store data.

2.2 Data Distribution

2.2.1 What Is a Chunk?

Chunks are sub‑ranges of data within a shard. They are split when they exceed the configured chunk size and balanced across shards.

2.2.2 Choosing Chunk Size

Typical chunk sizes are 100–200 MB. Smaller chunks improve balance but increase split overhead; larger chunks reduce splits but may cause uneven distribution.

2.2.3 Chunk Splitting and Migration

When a chunk exceeds the size limit, MongoDB splits it. The balancer then migrates chunks from overloaded shards to underloaded ones.

2.3 Sharding Keys

2.3.1 Shard Key

The shard key is an indexed field (or compound fields) that determines data distribution. It is immutable, must be present in every document, and is limited to 512 bytes.

2.3.2 Range‑Based Sharding

Data is partitioned based on ranges of the shard key.

2.3.3 Hash‑Based Sharding

MongoDB hashes the shard key value to distribute data evenly.

2.3.4 Shard Key Selection Advice

Increasing keys can cause write hotspots.

Random keys provide uniform distribution.

Hybrid keys combine benefits of both.

2.4 Deploying a Sharded Cluster

2.4.1 Environment Preparation

Create required directories for each shard instance.

2.4.2 Shard Configuration

Configure each shard as a replica set with clusterRole: shardsvr and start the mongod processes.

2.4.3 Config Server Setup

Configure three config server members as a replica set with clusterRole: configsvr and start them.

2.4.4 mongos Configuration

Configure mongos with the config server connection string and start it.

cat >/mongodb/28017/conf/mongos.conf<<'EOF'
systemLog:
  destination: file
  path: /mongodb/28017/log/mongos.log
  logAppend: true
net:
  bindIp: 10.0.0.152
  port: 28017
sharding:
  configDB: configReplSet/10.0.0.152:28018,10.0.0.152:28019,10.0.0.152:28020
processManagement:
  fork: true
EOF
mongos -f /mongodb/28017/conf/mongos.conf

2.4.5 Add Shards

db.runCommand({addshard: "sh1/10.0.0.152:28021,10.0.0.152:28022,10.0.0.152:28023", name: "shard1"});
db.runCommand({addshard: "sh2/10.0.0.152:28024,10.0.0.152:28025,10.0.0.152:28026", name: "shard2"});

2.4.6 Enable Database Sharding

sh.enableSharding("test");
sh.shardCollection("test.vast", {id: 1});

2.5 Sharded Cluster Operations

Use sh.status(), db.runCommand({listshards:1}), sh.enableBalancing() / sh.disableBalancing(), and other admin commands to monitor and manage the cluster.

2.6 Balancer Management

Check balancer state with sh.getBalancerState(). Stop it with sh.stopBalancer() or sh.setBalancerState(false). Start it with sh.startBalancer() or sh.setBalancerState(true). Configure active windows in the config database:

use config
db.settings.update({_id:"balancer"},{ $set: { activeWindow: { start: "00:00", stop: "05:00" } } }, {upsert:true});