Evolution of JD Baitiao Backend Architecture: From MySQL to ShardingSphere

JD Baitiao, a large‑scale internet finance product, has grown from a modest monolithic system to a massive backend serving billions of users, prompting continuous architectural evolution to meet rising data volume and performance demands.

1. Technology Lifespan: From MySQL to NoSQL to DBRep

There is no forever‑right technology; the best choice fits the current context.

2014‑2015: A Solr + HBase solution indexed searchable fields with Solr and stored full data in HBase, relieving core database pressure but introducing operational complexity.

Solr cluster offloaded part of read/write traffic.

Solr had poor extensibility and high business intrusion.

2015‑2016: Introduced MongoDB for monthly sharded tables, handling massive settlement data imports/exports with high query efficiency and flexible schema, yet suffered from poor scalability, high memory consumption, and strong business intrusion.

Hot‑data queries were fast; schema changes were easy.

Scaling limitations and high memory usage persisted.

2016‑2017: Data surpassed 100 billion rows; MongoDB faced capacity and performance bottlenecks. JD Baitiao adopted DBRep to capture MySQL slave changes, routing them to a message hub and finally persisting to Elasticsearch and HBase, achieving real‑time data flow with good scalability but increasing code coupling.

High real‑time performance and scalability.

Business‑level sharding increased code maintenance cost.

2. Decoupling the Backend for Seamless Scaling

To ensure a smooth shopping experience, backend database stability and regularity must be balanced.

The team identified four decoupling goals: data‑architecture decoupling, technical‑architecture decoupling, business‑relationship decoupling, and development‑process decoupling. They evaluated mature sharding middleware against custom solutions, requiring product maturity, extreme performance, massive data handling, and flexible extensibility.

Reduce coupling between architectures to avoid cascading changes.

Simplify upgrade processes and lower development effort.

Maintain user experience during large‑scale events (e.g., 618, 11.11).

Boost developer productivity.

3. Apache ShardingSphere as the Chosen Solution

ShardingSphere‑JDBC provides a lightweight, JDBC‑compatible driver that adds sharding capabilities without extra deployment. Its key features include product maturity, minimal performance overhead, low migration effort, and flexible extensibility.

Custom Sharding Framework

ShardingSphere

Performance

High

High

Code Coupling

High

Low

Business Intrusion

High

Low

Upgrade Difficulty

High

Low

Scalability

Average

Good

Ultimately, JD Baitiao selected Apache ShardingSphere for its financial‑grade data‑sharding capabilities.

4. Facing Unstable Business Scenarios with Stable Standards

How to understand instability and balance it?

As data importance grows and business lines proliferate, the industry experiences fragmented database ecosystems. JD Baitiao adopts the “Database Plus” concept—an intermediate layer that unifies data management, reduces operational costs, and offers plug‑in extensibility.

5. Returning to Fundamentals

For large‑scale domains like finance, the most effective solution is not always the newest technology but a mature middleware that incrementally enhances existing stacks, preserving stability while enabling growth.

Readers are encouraged to discuss, ask questions, and explore the detailed implementation in the referenced "Apache ShardingSphere – JD Baitiao Practical Guide".