Design and Evolution of Feed Stream Architecture for High‑Throughput Applications

Background – Feed streams (information flows) connect content producers and consumers in scenarios such as e‑commerce homepages, social circles, micro‑blogs, and news feeds. The author, a big‑data engineer involved in Alibaba's Taobao and Weitao feed storage services, shares insights from building HBase/Lindorm‑based storage for popular media platforms.

Business Analysis – Three participants (information producers, consumers, and the platform) have distinct needs: rich content formats, low latency, reliability, high engagement, and monetization. Feed delivery can be relationship‑based or recommendation‑based; the latter relies on user and item profiling.

Technical Challenges – Real‑world feed systems handle billions of daily messages, QPS in the millions, and require sub‑second response times, multi‑level caching, high availability, and cost‑effectiveness.

Mainstream Architecture

1. Message and Relationship Storage – Messages are semi‑structured (text, images, video, metadata). The author recommends HBase for its support of structured and semi‑structured data, high write throughput, smooth horizontal scaling, block cache for millisecond reads, TTL for data expiration, and hot‑cold data separation.

2. Push vs. Pull vs. Hybrid – Push copies a message to each follower’s inbox, offering low latency but high storage cost; pull reads from producers’ outboxes, requiring sophisticated caching. A hybrid approach combines push for active users and pull for inactive ones, reducing waste.

3. Recommendation‑Driven Delivery – Adds user/item profiling stored in HBase, a temporary inbox for filtering low‑quality content, and search capabilities (Solr/Elasticsearch) for both inbox and outbox.

Iteration Path for Start‑ups

Initial stage: cloud Kafka + cloud HBase, pure push, asynchronous processing, message split into body and index.

Mid‑stage: add multi‑level queues to handle “big‑V” traffic spikes, hot‑cold data separation, and secondary indexes for inbox search.

Growth stage: adopt push‑pull hybrid, integrate recommendation system, use cloud Kafka + cloud HBase + cloud Redis, and enhance outbox caching and inbox secondary indexing.

Conclusion – Feed streams are ubiquitous across e‑commerce, social, and media apps. The article summarizes business scenarios, mainstream architectures, and technical bottlenecks, and provides a concrete cloud‑native evolution roadmap. Ongoing challenges include handling inactive accounts, reducing storage cost, and improving search, which will drive further optimization of cloud HBase for feed workloads.