Why MongoDB 4.0 Increases CPU Load After Upgrade – Balancer and moveChunk Explained

Problem Overview

In a production three‑shard cluster the upgrade from MongoDB 3.2 to 4.0 caused CPU load to rise dramatically (from ~10 % to ~40 %). No changes were made to application logic or traffic volume. Closing the balancer restored the load to normal levels, indicating the balancer’s activity as the root cause.

Monitoring and Log Evidence

During periods when the balancer was enabled, mongostat showed a spike in CPU usage that matched a surge in delete operations. The logs contain entries such as:

53094:2019-10-08T10:09:24.035199+08:00 I SHARDING [Collection Range Deleter] No documents remain to delete in dt2log.tbl_log_item_20191001 range [{ _id: -3074457345618258602 }, { _id: -3033667061349287050 })
53095:2019-10-08T10:09:24.035222+08:00 I SHARDING [Collection Range Deleter] Waiting for majority replication of local deletions in dt2log.tbl_log_item_20191001 range [{ _id: -3074457345618258602 }, { _id: -3033667061349287050 })
53096:2019-10-08T10:09:24.035274+08:00 I SHARDING [Collection Range Deleter] Finished deleting documents in dt2log.tbl_log_item_20191001 range [{ _id: -3074457345618258602 }, { _id: -3033667061349287050 })

The correlation between the delete bursts and CPU spikes confirms that post‑migration deletions are the primary load driver.

splitChunk, Balancer, and moveChunk Mechanics

When a client performs inserts, updates, or deletes, mongos estimates the size of the affected chunk. If the estimated size exceeds maxChunkSize/5, a splitChunk is triggered via the splitVector command. After a split, the balancer checks data distribution; if a shard holds significantly more chunks than another, it initiates a moveChunk task. The source shard then asynchronously deletes the migrated documents.

The fundamental moveChunk flow is identical in 3.2 and 4.0, but the location of the balancer changed: in 3.2 it runs on mongos, while from 3.4 onward it runs on the config servers.

Imbalance Threshold Differences

Version 3.2 uses a relatively lax threshold:

int threshold = 8;
if (balancedLastTime || distribution.totalChunks() < 20) threshold = 2;
else if (distribution.totalChunks() < 80) threshold = 4;

Version 4.0 adopts a much stricter rule—any chunk count difference greater than 2 triggers balancing:

const size_t kDefaultImbalanceThreshold = 2;
const size_t kAggressiveImbalanceThreshold = 1;
size_t imbalanceThreshold = (shouldAggressivelyBalance || distribution.totalChunks() < 20)
    ? kAggressiveImbalanceThreshold : kDefaultImbalanceThreshold;

Consequently, a collection with 47 chunks on one shard and 53 on the other is considered balanced in 3.2 (no moveChunk), but unbalanced in 4.0 (moveChunk occurs).

moveChunk Configuration Parameters

Two key settings influence the migration workload:

_secondaryThrottle : when true, the balancer waits for acknowledgment from at least one secondary before proceeding. Default is true in 3.2 and false from 3.4 onward.

waitForDelete : when true, the balancer waits synchronously for the orphaned chunk data to be deleted. Default is false.

These can be set via the config.settings collection:

use config
db.settings.update(
  { "_id" : "balancer" },
  { $set : { "_secondaryThrottle" : { "w": "majority" }, "_waitForDelete" : true } },
  { upsert : true }
)

Additional Parameters Affecting Migration Speed

migrateCloneInsertionBatchDelayMS : delay between successive insert batches during migration (default 0 ms).

migrateCloneInsertionBatchSize : number of documents inserted per batch (default 0, meaning unlimited).

They can be adjusted with admin commands:

db.adminCommand({setParameter:1, migrateCloneInsertionBatchDelayMS:0})
db.adminCommand({setParameter:1, migrateCloneInsertionBatchSize:0})

Asynchronous Delete Thread

Both versions use a background thread to clean up ranges after a moveChunk, but the implementation differs:

3.2: db/range_deleter.cpp – entry point RangeDeleter::doWork().

4.0: db/s/metadata_manager.cpp – cleanup scheduled via scheduleCleanup.

Deletion is performed in batches; the batch size is determined by rangeDeleterBatchSize or, if zero, by internalQueryExecYieldIterations (default 128):

maxToDelete = rangeDeleterBatchSize.load();
if (maxToDelete <= 0) {
    maxToDelete = std::max(int(internalQueryExecYieldIterations.load()), 1);
}

Other tunable parameters include rangeDeleterBatchDelayMS (default 20 ms), internalQueryExecYieldIterations (default 128), and orphanCleanupDelaySecs (default 900 s).

Performance Impact Summary

moveChunk’s delete phase is the main source of CPU load; the insert side has minimal impact.

From 3.4 onward the balancer’s imbalance threshold is lower, causing more frequent moveChunk operations.

Collections sharded on a hashed key with many small documents should keep the balancer disabled after an initial chunk distribution.

Version‑specific defaults ( _secondaryThrottle, threshold logic) explain why 4.0 exhibits higher load than 3.2 under identical workloads.