How to Seamlessly Migrate MongoDB to New Data Stores Without Downtime

Current Situation

The production system stores all core business data in a MongoDB replica set (1 primary, 1 secondary) consisting of a single database with two collections. Seven applications depend on this data. The migration goal is to replace MongoDB with alternative storage back‑ends while keeping the service online.

Migration Plan

Migration Rhythm

Identify every business function that reads or writes MongoDB data. The system also contains MySQL sources, so the scope must include all MongoDB‑dependent modules.

Define the target storage media (JImKV, MySQL, Elasticsearch) and the performance targets required to match the existing read/write load.

Refactor the Data Access Object (DAO) layer to abstract the concrete data source. All direct MongoDB driver calls are moved into DAO interfaces without changing method signatures.

Introduce a dual‑write mechanism that writes to both the legacy MongoDB and the new store during the transition period.

Validate the migration by replaying production traffic (R2 flow), running functional tests, and performing data‑level comparisons.

Gradually shift traffic according to a controlled ramp‑up schedule.

Code Refactor & Data Heterogeneity

A decorator pattern wraps the DAO implementations. The decorator handles:

Primary write to the chosen new store.

Secondary write to the legacy MongoDB (or vice‑versa) for synchronization.

Traffic‑shaping logic that decides which store receives reads during gray‑release.

Fallback logic for write failures.

By keeping the original method signatures, existing business code remains untouched while the underlying storage can be swapped.

Bulk Data Migration

The total volume is about 3 million rows per table, which makes a pure offline batch job inefficient. A code‑driven migration splits the data into two categories:

Approved applications – data is static after approval; these records can be migrated at any time and verified against the source.

In‑process applications – data may change; these records are migrated during off‑peak hours with dual‑write enabled to keep source and target consistent.

The migration logic runs in a loop that fetches a batch of documents, transforms them to the target schema, writes to the new store, and records success/failure for later reconciliation.

Incremental Data Synchronization

For new or updated records that lack a status field, the write path is:

writeToMongoDB(record) // source of truth
if (mongoWriteSuccess) {
    writeToMySQL(record) // secondary store
    if (!mysqlWriteSuccess) {
        // send compensation message to MQ for asynchronous retry
    }
}

MongoDB success determines transaction commit; MySQL failures are compensated asynchronously via a message queue.

Deployment Strategy (Monitoring, Gray Release, Rollback)

Monitoring (Data Comparison)

Incremental comparison : After each dual‑write, an MQ message triggers a real‑time fetch of the same key from both stores, normalizes the objects, and logs any field mismatches. Alerts are raised for critical differences.

Bulk comparison : A scheduled job iterates over all records in the legacy MongoDB, retrieves the corresponding record from the new store, normalizes both representations, and writes any inconsistencies to a log file for offline analysis.

Gray Release (Traffic Shaping)

Read traffic is split based on a whitelist of merchants and a configurable percentage. A ThreadLocal variable stores the merchant identifier so that the routing decision is available across threads during a request.

Rollback (Write Path)

The cut‑over consists of four steps:

Validate that writes to the new store succeed under a gray‑release of the updated code. If failures are observed, revert writes to the legacy store.

Back‑fill any missing records in the new store identified during validation.

Switch the primary write target to the new store (remove the dual‑write decorator).

After confirming stable reads and writes, decommission the legacy MongoDB deployment.