How to Evolve Your Database Architecture for Massive Scale: From Monolith to Sharding

1 Introduction

As a system grows with expanding business, data volume increases dramatically; the key goal is to keep strong stability and high performance even when data becomes massive.

2 Evolution

2.1 Monolithic

Usually adopted at the initial product design or trial stage.

The most common architecture consists of:

Service: provides API (REST/RPC) to callers and accesses the database.

DB: a single database handling data storage.

2.2 Cold‑Hot Separation

Historical data with low access frequency can be placed in separate tables or databases, while recent hot data stays in a lightweight hot database for better performance.

Architecture:

Service: provides API and accesses databases.

DB‑Cool: stores large, historical data where slower loading is acceptable.

DB‑Hot: stores recent active data with high read/write efficiency.

2.3 Read‑Write Separation

When read operations far outnumber writes, read‑write separation is designed to solve high‑concurrency database access.

Typical 1‑master‑N‑slave architecture:

Service: provides API and accesses databases.

DB‑M (master): handles write operations.

DB‑R (replica): handles read operations.

Characteristics:

1 master N slaves linearly improve read performance, though too many slaves increase replication pressure.

Master and slaves replicate data via binlog.

Data structures and results stay consistent; eventual consistency is acceptable despite replication lag.

Writes remain single‑point because high‑frequency hot writes are usually a small proportion of traffic.

2.4 Data Sharding Architecture

Typical horizontal sharding (splitting) solution. Horizontal splitting can be done as table‑level or database‑level sharding; the article focuses on database‑level sharding.

Architecture:

Service: provides API and accesses databases.

DB‑0, DB‑2: data is partitioned into shards; the example stores 10 KB per shard.

Horizontal partitioning can use five strategies:

2.4.1 Hash

This strategy hashes one or more columns to determine the partition, e.g., partitioning by year of a timestamp.

<code>PARTITION BY HASH(YEAR(createtime))
PARTITIONS 10
</code>

2.4.2 Range

This strategy divides data into ranges, e.g., splitting a table of tens of millions of rows into four partitions based on id.

<code>PARTITION BY RANGE(id) (
  PARTITION p0 VALUES LESS THAN (2500001),
  PARTITION p1 VALUES LESS THAN (5000001),
  PARTITION p2 VALUES LESS THAN (7500001),
  PARTITION p3 VALUES LESS THAN MAXVALUE
)
</code>

2.4.3 What problems does it solve?

Database/table slimming to improve read/write performance.

Reduce the impact radius of failures.

3 Summary

Use a single‑database model in the early product stage.

Apply cold‑hot separation when part of the data becomes less frequently accessed.

Adopt read‑write separation (master‑slave) as read traffic dominates.

Introduce sharding (database/table splitting) when data volume overwhelms single‑node performance.

And then go for a run!