Mastering Database Sharding: Strategies for Scaling High‑Traffic Applications

Chapter 1 Introduction

With the widespread adoption of internet applications, massive data storage and access have become bottlenecks in system design. A large‑scale web service handling billions of page views per day puts enormous load on databases, threatening stability and scalability. Horizontal data partitioning (sharding) is the preferred method for improving performance and enabling scale‑out.

Horizontal sharding reduces load on each machine and minimizes loss from failures.

Load‑balancing strategies lower per‑machine access pressure and reduce outage probability.

Cluster solutions eliminate single‑point failures of databases.

Read‑write separation maximizes read speed and concurrency.

Chapter 2 Basic Principles and Concepts

What Is Data Sharding

The term “shard” means “fragment”. In database terminology, sharding (horizontal partitioning) distributes data across multiple databases or tables to overcome the I/O limits of a single node and improve scalability. Data is split according to rules, and routing logic directs queries to the appropriate database or table.

For example, a blog’s article table can be sharded by user_id: IDs 1‑10,000 go to DB1, 10,001‑20,000 to DB2, and so on. The routing process uses the same user_id to locate the target database.

Sharding often introduces redundant fields (e.g., user_id) that serve as partition keys, which may violate strict normalization but are necessary for efficient horizontal scaling.

Why Sharding Is Needed

Although mature databases like Oracle can handle large volumes, their high licensing and hardware costs make them unsuitable for many companies. When a single node reaches its physical limits, horizontal expansion by adding more machines becomes the practical solution.

Sharding distributes compute, storage, and I/O across multiple servers, improves fault isolation, and avoids single‑point failures, thereby reducing operational costs while achieving performance comparable to expensive enterprise solutions.

How to Implement Sharding

Physical sharding distributes data across different DB servers using routing rules, reducing load per machine.

Logical sharding within a database splits a large table into multiple sub‑tables (e.g., article_001, article_002), decreasing index size and insert latency.

Common partitioning strategies include:

Range partitioning : Assign ID ranges to specific databases (e.g., 1‑1000 → DB1). Simple to migrate but may cause uneven data distribution.

Hash modulo partitioning : Hash the partition key and take modulo N to select a database, yielding uniform distribution but complicating data migration.

Mapping table : Store a separate mapping DB that records which key maps to which database, offering flexibility at the cost of an extra lookup.

Complex projects often combine these methods.

Chapter 3 Research Outline

The distributed data solution provides:

Database and routing rules (RouteRule).

Cluster (Group) concept for high availability.

Load‑balancing policies (LB).

Node health‑checking to keep LB decisions accurate.

Read/write separation to boost query speed.

Even with sharding, a single node failure still makes part of the data unavailable. Introducing a Group—multiple replicas of each shard—ensures that if one replica fails, others can serve traffic, greatly improving fault tolerance.

Each Group contains one Master (or multiple) handling writes and several Slaves handling reads. Load balancers direct write traffic to Masters and read traffic to Slaves based on policies such as random or weighted distribution, the latter accounting for differing server capacities.

To avoid routing to failed nodes, the system employs either active health‑checking (periodic connection attempts) or passive status push from DBAs, ensuring the load balancer only uses healthy replicas.

Read/write separation leverages the typical 10:1 read‑to‑write ratio in internet services, concentrating writes on Masters to avoid lock contention and distributing reads across Slaves for higher throughput.