How to Optimize a Billion‑Scale MongoDB Cluster for Sub‑Second Latency on Limited Hardware

Background: An OPPO‑managed MongoDB service stores about 100 billion documents (≈7.5 TB raw, 2.5‑3 TB after compression) and keeps only the most recent seven days. Peak write traffic reaches 140 k ops/s, causing a concentrated deletion load when data expires, which creates a performance bottleneck.

Resource Evaluation and Deployment Architecture

Data volume: 100 billion records, each ~800 bytes → ~7.5 TB total.

Read‑write separation, 7‑day retention.

Memory per container: 64 GB (based on historic 64 GB limits for similar clusters).

Sharding: three shards required to handle 10 W/s peak writes; each shard allocated 1 TB storage.

CPU: limited to 16 CPU cores per shard container.

mongos and config server: combined into a single container with 8 CPU, 8 GB RAM, 50 GB disk.

Deployment uses a 2+1 pattern (two mongod nodes + one arbiter) within the same data center to minimize cost.

Performance Optimization Process

Pre‑deployment Optimizations

Create an index on createTime with expireAfterSeconds: 604800 (7 days).

Choose userId as the shard key and use hashed sharding to evenly distribute data while keeping a user’s records on the same shard.

Enable pre‑splitting with 8192 initial chunks to avoid chunk migrations.

Configure clients to use secondaryPreferred for near reads.

Disable enableMajorityReadConcern because the application does not require strict majority reads.

Set WiredTiger cache size to 42 GB (on a 64 GB container) to prevent OOM during large migrations.

First‑Round Storage Engine Tuning

Increase WiredTiger eviction thread count to handle continuous high‑write load:

db.adminCommand({ setParameter: 1, "wiredTigerEngineRuntimeConfig": "eviction=(threads_min=4, threads_max=20)" })

This mitigates write stalls caused by excessive dirty‑page eviction.

Issues After First Round

When data expires after a week, delete spikes combine with ongoing writes, pushing total ops to ~20 W/s, which exceeds the capacity of three shards. Monitoring shows low write counts per primary (≈4 k ops/s) but a persistent >20% dirty‑page ratio, leading to request latency spikes up to several hundred milliseconds.

Second‑Round Attempts

Moving TTL deletions to a low‑traffic window proved ineffective because nightly batch reads overlapped with deletions, still causing latency spikes.

Third‑Round Solution: Daily Table Partitioning

Rewrite the application to create a separate collection for each day (e.g., daily_20230614).

Enable pre‑splitting on each daily collection to keep data balanced across the three shards.

Retain data for eight days; at 00:00 on the ninth day, drop the oldest daily collection instead of relying on TTL.

This approach eliminates massive delete bursts, keeping write/read latency stable.

Optimization Results

Before optimization: latency frequently spiked to several hundred milliseconds.

After optimization: latency stabilized between 0.5 ms and 2 ms, with no need for additional hardware.

All changes were implemented within the existing three‑shard cluster, achieving significant performance gains while minimizing cost.