Evolution of Weibo Recommendation Architecture: From Independent 1.0 to Platform 3.0

Introduction

Weibo is a broadcast‑style social network where users follow others to receive short, real‑time information. In this context, the recommendation system can be tightly integrated with the subscription‑distribution mechanism, mutually reinforcing each other.

Weibo's two core foundations are user‑relationship construction and content propagation; the recommendation system continuously optimizes both to promote platform growth.

The recommendation workflow, common across the industry, consists of candidate generation, ranking, strategy, presentation, feedback, and evaluation, forming a closed loop.

1. Independent 1.0

1.1 Environment

The architecture emerged between July 2011 and February 2013 to meet immediate business needs. The team was new, with limited collaboration and little shared experience in recommendation systems. External pressures—rapid product growth, many parallel projects, and short development cycles—forced the use of familiar technology stacks for each project, resulting in isolated architectures.

Small, inexperienced team with fragmented cooperation.

Multiple concurrent projects (3‑5 per 5‑person team) driven by the fast‑growing Weibo product.

External teams supplied most requirements, leading to scattered effort.

Industry was still exploring recommendation architectures.

1.2 Architecture Composition and Characteristics

Technical goals focused on delivering business functionality without a complete feedback or evaluation loop; the pipeline was essentially candidate → strategy → simple presentation.

Key technology stack:

Web service: Apache + mod_python (later mod_wsgi) using Python for rapid development.

Computation service: C/C++ based "woo" framework.

Databases: Redis, custom mapdb, keylistdb for static and dynamic storage.

Data source: Rsync file transfer and internal firehose queue.

Advantages: simple, easy to implement, supports rapid business feature delivery, and allows parallel development without interference.

Disadvantages: incomplete recommendation loop, lack of unified data handling, no algorithmic support, difficult to perform professional operations and QA, and poor team collaboration.

1.3 Outcomes

Supported over twenty independent projects during Weibo's rapid growth.

Created the foundational "woo" computation framework.

Developed mapdb for static storage, later becoming the basis for static data handling.

Established a common recommendation application framework.

2. Layered 2.0

2.1 Environment

From March 2013 to the end of 2014, the team had matured, achieving better internal collaboration and consensus on technology choices. The product focus shifted to three recommendation scenarios (feed, article page, PC homepage). Externally, the company clarified the role of recommendation for relationship building, content dissemination, and ad support, while industry best practices provided guidance.

2.2 Architecture Composition and Characteristics

Technical goals expanded to cover the full recommendation pipeline (candidate, ranking, strategy, presentation, feedback, evaluation), data‑first principles, and algorithmic integration while maintaining rapid business iteration.

Key layers:

Application layer : Implements recommendation strategies and presentation, built on an extended Apache + mod_python framework called common_recom_frame, which abstracts project, work, and data interfaces.

Computation layer : Handles ranking calculations using the C/C++ "WOO" framework (extended with project/work/data management). Open‑source implementation: lab_common_so .

Data layer : Manages IN/OUT/STORE of static and dynamic data. Static data flows from Hadoop (MR, Hive, Spark) via R9‑interface; dynamic data enters via RIN (web service + ckestrel queue). Storage uses Redis for dynamic data and Lushan cluster for static data (open‑source: lushan ). tmproxy/gout handles data OUT.

Infrastructure services : Monitoring, alerting, and evaluation systems (performance and effect monitoring).

Advantages: complete recommendation loop, unified data handling, strong algorithm support, data‑driven effect improvement, and easier deployment and QA.

Shortcomings: still not fully tailored to recommendation core, algorithm training not covered, and strategy logic remains developer‑driven.

2.3 Outcomes

Core recommendation products (article page, trending users/content, fan economy, etc.) operated within this architecture.

Open‑sourced lab_common_so framework.

Open‑sourced static storage solution lushan.

Developed the RUF framework, contributing to the OpenResty community.

3. Platform‑Style 3.0

3.1 Environment

From late 2014 to present, internal focus shifted from rapid feature expansion to effect‑oriented technical iteration. Repeated work across projects highlighted the need for a unified architecture.

Externally, the company prioritized efficiency, user experience, and content quality, while the recommendation field offered opportunities to catch up with industry advances.

3.2 Architecture Composition and Characteristics

Technical goals now emphasize abstracting common methods for candidate generation, ranking, training, and feedback, and aligning the system closely with algorithmic needs.

Key differences from 2.0:

Standardized interfaces for the application layer (all‑in‑one I/O) and dynamic input layer (RIN).

Added candidate generation modules (Artemis, item‑cands, etc.) in the computation layer.

Introduced the EROS strategy platform for model training, feature selection, and online A/B testing.

Extended data layer interfaces (R9‑interface, RIN) to support unified candidate generation.

Advantages: inherits 2.0 strengths, deeper algorithmic integration, and resolves key issues in candidate, ranking, and training.

3.3 Outcomes

Core recommendation business gradually migrated to this platform, driven by algorithmic data.

Established the EROS training pipeline with standardized methods.

Defined standard input/output protocols for recommendation services.

Created an abstract collection of candidate generation methods.

4. Summary

The evolution of Weibo's recommendation architecture demonstrates a strong feedback loop between business needs and technical solutions: business drives technology, and improved technology further accelerates business growth.

Key takeaways include: seek the shortest viable path and iteratively optimize; promote frameworks through community participation rather than mandates; and focus on clear goals, data, and metrics to align product, architecture, and algorithms.

— THE END —