Analysis of MongoDB 3.2 vs 4.0 Balancer and moveChunk Impact on CPU Load

Problem

After upgrading a three‑shard MongoDB cluster from version 3.2 to 4.0, CPU load rose dramatically from about 10% to 40% even though the application logic and traffic remained unchanged. Closing the balancer returned the load to normal, but reopening it on Tuesdays caused the load to stay high.

Monitoring and Logs

Monitoring showed that the CPU spike correlated with the period when the balancer was active and with a large number of delete operations. The logs contain entries such as:

53094:2019-10-08T10:09:24.035199+08:00 I SHARDING [Collection Range Deleter] No documents remain to delete in dt2log.tbl_log_item_20191001 range [{ _id: -3074457345618258602 }, { _id: -3033667061349287050 })
53095:2019-10-08T10:09:24.035222+08:00 I SHARDING [Collection Range Deleter] Waiting for majority replication of local deletions in dt2log.tbl_log_item_20191001 range [{ _id: -3074457345618258602 }, { _id: -3033667061349287050 })
53096:2019-10-08T10:09:24.035274+08:00 I SHARDING [Collection Range Deleter] Finished deleting documents in dt2log.tbl_log_item_20191001 range [{ _id: -3074457345618258602 }, { _id: -3033667061349287050 })

The high CPU is therefore mainly caused by the massive deletions that follow chunk migrations. The collections use {_id : hashed} sharding with many small documents (average >100k docs per chunk), so deleting a migrated chunk can take around ten minutes.

splitChunk

When inserts, updates, or deletes occur, mongos estimates the chunk size and may trigger splitChunk . The estimation is coarse, leading to delayed splits and temporary imbalance that can cause extra moveChunk operations.

Initial chunk size is a random value between 0 and 1/5 of maxChunkSize .

Each write adds the document size to the estimated chunk size.

When the estimate exceeds maxChunkSize/5 , splitVector is called; if it returns split points, splitChunk is executed.

Balancer

Both 3.2 and 4.0 run the balancer every 10 seconds (1 second after a migration is detected). The balancer reads shard info from config.shards , collection info from config.collections , and chunk info from config.chunks . It selects a source shard with the most chunks and a destination shard with the fewest; if the difference exceeds a threshold, a moveChunk task is created.

The threshold logic differs between versions:

3.2: threshold = 2 for <20 chunks, 4 for 20‑79 chunks, 8 for ≥80 chunks.

4.0: any difference greater than 2 triggers migration, making the balancer more aggressive.

Consequently, the 4.0 cluster performs many more moveChunk operations, leading to higher CPU usage.

moveChunk Details

During a moveChunk , the destination shard first deletes the incoming chunk’s data. Parameters such as _secondaryThrottle and waitForDelete can be tuned via config.settings :

use config
 db.settings.update(
   { "_id" : "balancer" },
   { $set : { "_secondaryThrottle" : { "w" : "majority" }, "_waitForDelete" : true } },
   { upsert : true }
 )

In 3.2 the default _secondaryThrottle is true ; from 3.4 onward it defaults to false , so 4.0 completes migrations faster and inserts more data on the destination shard, adding to CPU load.

Additional parameters that affect migration speed include migrateCloneInsertionBatchDelayMS , migrateCloneInsertionBatchSize , rangeDeleterBatchDelayMS , internalQueryExecYieldIterations , rangeDeleterBatchSize , and orphanCleanupDelaySecs . For example:

db.adminCommand({setParameter:1, migrateCloneInsertionBatchDelayMS:0})
 db.adminCommand({setParameter:1, migrateCloneInsertionBatchSize:0})

The asynchronous deletion thread also changed: 3.2 uses db/range_deleter.cpp (function RangeDeleter::doWork() ), while 4.0 uses db/s/metadata_manager.cpp with batch deletions controlled by rangeDeleterBatchSize and rangeDeleterBatchDelayMS .

Conclusion

moveChunk can significantly increase system load, primarily during the deletion phase.

Versions 3.4 and later have lower balancer thresholds, causing more frequent migrations.

For collections with many small, hashed‑sharded documents, it is advisable to pre‑allocate chunks and keep the balancer disabled; otherwise, 4.0’s aggressive balancing leads to higher CPU consumption compared with 3.2.