Design Considerations for a Scalable High‑Availability Weibo Distributed Storage System

Recently, the "High Availability Architecture" series posted a Weibo distributed storage exam question, prompting many engineers to submit their solutions; this article adds further clarifications for architects discussing implementation.

Access Scenarios : Users mainly view their latest received posts and browse a specific user's profile, requiring retrieval of the most recent n posts from followed users and paginated access to a user's historical posts.

Access Characteristics : Over 90% of accesses target data from the last seven days.

Points Not Required :

No cache layer design is needed.

Write peak handling can be ignored due to asynchronous writes.

ID generation is assumed to be provided by an external service.

Aggregation of received posts is handled by the service layer.

Design Considerations

Scale‑out : Split data across multiple independent storage units, support re‑sharding with online access, and consider cost by using inexpensive hardware rather than assuming all‑SSD storage.

Cost : Store data with different access speeds on hardware of varying cost.

High Availability & Reliability : Use MySQL replication to ensure availability and read‑write separation.

Sharding Strategies

Range‑based: Group adjacent user IDs together.

Hash‑based: Distribute a user's data to a specific shard using a hash function.

Re‑Sharding Design : When data growth exceeds node capacity, further split nodes while maintaining online access.

Capacity Planning :

Pre‑plan to accommodate future data volume.

Use powers of two for shard count to simplify scaling.

Trade‑offs :

Many shards may degrade I/O efficiency.

Few shards lead to frequent and complex scaling operations.

Case Studies

Case 1: Use user‑ID range as shard.

Case 2: Use user‑ID hash as shard.

Case 3: Additional example by another author.

Historical Data :

Every six months, create a separate database (e.g., 2015.01‑2015.06). With 100 million rows per day, six months produce ~180 billion rows (~0.72 TB), fitting within a 1 TB disk.

Shard by UID modulo to distribute tables and facilitate queries.

Current N‑Day Data (n≈10, ~1 billion rows):

Store recent data based on UID plus a weighted hash; weight derived offline from post count, follower count, etc.

Prepare for traffic spikes by employing two‑ or three‑level sharding, using a flag in ZooKeeper to select hash algorithms with time‑based dimensions.

The hash algorithm must ensure that the same UID maps to a single database and that users with similar weights are evenly distributed across different databases.

Query Indexing :

Add a posting index field to record each user's post identifiers.

Maintain a total post count per user, incremented on each new post.

Create a secondary index table recording, for each user, the position of a post within a specific shard (e.g., the 10th post of 2015 for UID 1).

Use the secondary index for pagination: first locate the relevant real database(s), then fetch the data.

Summary :

Time‑based sharding avoids massive migrations caused by UID‑based shard count limits and enables hot‑cold data separation.

Shard logic can be flexibly controlled via ZooKeeper flags (e.g., month‑level or year‑level partitions).

Offline weight calculation distributes load evenly; weight adjustments affect only a small subset of users, minimizing migration impact.

Secondary indexing reduces data volume; modifying B‑tree structures can further optimize but is complex.

Integrate recent n‑day data with current day data for unified access.

Further Reading : For details on pagination techniques for extremely long lists, see the linked original article.

Interested readers can submit their design solutions to [email protected] and follow the "High Availability Architecture" WeChat public account for more topics; please credit the ArchNotes WeChat public account when reproducing this content.