Scalable Feed Stream System Design: Architecture, Storage & Sync

Introduction

Feed streams are continuous flows of information units (posts, videos, status updates) that users consume on mobile devices. They are built on the concept of constantly delivering fresh "feed" items from many publishers to many receivers.

Data Model

Publisher data – the content generated by each user.

Follow relationships – one‑way (e.g., Weibo) or two‑way (e.g., WeChat Moments) connections that determine who receives which feeds.

Receiver data – ordered by time or score, representing each user’s personal timeline.

Storage Design

The system needs three logical stores:

Repository: permanent storage of publisher data.

Follow table: persistent user‑relationship data.

Sync store: short‑term cache of recent receiver data.

For large‑scale deployments, distributed NoSQL (Tablestore, Bigtable) is preferred for its horizontal scalability and high reliability; smaller systems may start with MySQL.

Synchronization Modes

Push (write‑fan‑out) : publisher writes once and the message is duplicated to each follower’s inbox; requires very high write throughput.

Pull (read‑fan‑out) : followers read from the publisher’s outbox on demand; high read load and complex offset tracking.

Push‑Pull hybrid : normal users use push, while high‑profile "big V" users use pull to reduce wasted writes.

Metadata

Additional tables support the core feed functionality:

User profile and list tables.

Follow/friend relationship tables.

Push session pool to track online users and avoid query storms.

Comments and Likes

Both are stored similarly to feed items, with an extra reference to the parent feed. Distributed NoSQL is ideal because the operations are simple inserts and range queries.

Search

Simple keyword search for users, posts, or friends can be implemented via a search engine or a full‑text capable database (MySQL, MongoDB, Tablestore). Adding a multi‑field index on the relevant tables is sufficient.

Ordering

Two primary ordering strategies exist: chronological (time‑line) and score‑based (personalized recommendation). Time‑line ordering suits follower‑driven feeds; score‑based ordering powers recommendation‑driven feeds like Toutiao.

Deletion & Update

Deletion can be physical (remove from repository) or logical (set a delete flag). Updates follow the same path; versioned stores such as Tablestore allow multi‑version edits.

Overall Architecture Choices

Two viable approaches:

Single‑product solution using Alibaba Cloud Tablestore, which provides repository, follow, sync, and index capabilities in one service.

Open‑source combo (MySQL, Redis, HBase, etc.) requiring careful integration but offering flexibility for teams comfortable with operations.

Practical Scenarios

WeChat Moments – two‑way relationships, time‑line ordering, push mode.

Weibo – one‑way relationships, big‑V effect, hybrid sync.

Toutiao – recommendation‑driven, score ordering, push‑pull hybrid.

Private messages – simple one‑to‑one feed, push mode.

Conclusion

The article outlines the complete design theory for feed‑stream systems and provides concrete guidance on storage selection, synchronization strategies, metadata handling, search integration, and scaling to billions of users.