How to Build a Scalable Feed Stream: Architecture, Models, and Best Practices

What Is a Feed Stream?

A feed stream is a continuously updated list of items shown to users, aggregating content from multiple sources (e.g., friends, followed accounts, recommendation engines) into a personalized timeline. Typical examples include the "Friends Circle" in WeChat, the follow page in Weibo, and the recommendation page in Douyin.

Why Feed Streams Exist

Traditional media (TV, newspapers, magazines) require users to actively seek information, which is inefficient for modern consumption. Feed streams solve this by aggregating relevant content based on user behavior or relationships, delivering it instantly as users scroll, thus improving the user experience.

Classification of Feed Streams

Feed streams can be grouped by the aggregation logic:

Interest‑based recommendation – e.g., Douyin recommendation page, where algorithms rank items by weight.

Relationship‑based timeline – e.g., Weibo follow page or WeChat friends circle, where items are ordered chronologically based on follow relationships.

Each classification serves different business scenarios such as content discovery, subscription, or social networking.

Historical Evolution

Feed streams originated from RSS aggregators, which combined user‑subscribed feeds into a timeline. Facebook’s News Feed transformed the model by aggregating content from people rather than sites, leading to richer, socially driven feeds.

Key Terminology

Feed : a single status or message (e.g., a WeChat post).

Feed Stream : the overall data flow that pushes multiple publishers’ content to many receivers.

Timeline : a type of feed stream that presents items in chronological order.

Write Diffusion (Push) : the publisher writes the new message ID directly into each follower’s inbox.

Read Diffusion (Pull) : the follower pulls messages from each publisher’s outbox when needed.

Challenges in Feed‑Stream Modeling

Real‑time performance: messages are generated, consumed, and pushed instantly, demanding low latency.

Massive scale: billions of messages require efficient storage and retrieval.

Read‑write imbalance: far more reads than writes, requiring careful sharding and indexing.

Consistency: the system must guarantee at‑least‑once delivery and avoid message loss.

Functional Requirements

User can publish a message; followers see it instantly.

User can delete a message; followers see the deletion.

User can view their own published messages.

User can subscribe/unsubscribe to other users, with optional back‑fill of historic messages.

Support black‑/white‑list permissions and comment CRUD operations.

Solution Overview

Because feed streams are read‑heavy, the default strategy is write diffusion (push) for normal users. For high‑profile users ("big V"), a hybrid read‑write combined approach is used: push to active followers, pull from inactive followers, and apply hot‑cold separation based on activity metrics.

Pagination Strategy

Instead of page number/size, the API uses last_id (the ID of the last item from the previous page). The backend fetches the next page_size items after that ID. Deletions are handled via a soft‑delete flag; if a deleted item appears, the service skips it and fetches additional rows.

Write‑Diffusion Pagination

Since each follower’s inbox (a sorted list) is pre‑populated, pagination simply reads the next range after last_id.

Read‑Diffusion Pagination

Followers pull from each publisher’s outbox, maintaining a per‑publisher cursor ( write_last_id). The service merges the fetched slices, filters deleted items, and returns a unified page.

Overall Architecture Design

The system consists of:

A message queue that receives new feed items.

Publisher service that writes the message to the author’s timeline (outbox).

Follower service that pushes the message ID to each follower’s inbox (Redis ZSET) or records it for later pull.

Storage layer with three main tables: Message (msg_id, title, content, type, status, channel, timestamps), Inbox (Redis ZSET keyed by uid:channel storing sender_id+msg_id with timestamp scores), FollowRelation (main_uid, follower_uid, status, hot_follower flag).

Additional tables for PublishConfig (send_type, cron, channel) support future push‑notification extensions.

Core Business Flow

Publish Feed Process

Publisher sends a new feed message to the queue.

Service retrieves the author’s follower list and determines if the author is a big V.

Message is written to the author’s own timeline (outbox).

For normal users, the message ID is pushed to every follower’s inbox; for big V, only active followers receive a push, while inactive followers will pull later.

Read Feed Process

Determine if the reader is active; if not, fetch the list of followed big V users.

Read the reader’s inbox from last_id forward, then resolve each ID to full message content, filtering soft‑deleted items.

If big V followees exist, concurrently pull their outboxes and merge results.

Merge, sort by timestamp, and return the page to the client.

Summary

The article provides a complete blueprint for building a reliable, high‑performance feed stream system, covering the conceptual background, classification, data modeling, storage choices (MySQL + Redis), pagination technique, diffusion strategies, and end‑to‑end publish/read workflows. Engineers can adapt this design to their own products, adjusting hot‑cold separation, diffusion mode, and table schemas as needed.