How Weibo’s Recommendation Engine Evolved: From 1.0 to Platform‑Scale 3.0

Introduction

Wang Chuanpeng, former director of recommendation and advertising technology at Sina Weibo, presents a comprehensive overview of the evolution of Weibo's recommendation system, highlighting how business needs and technical advances shaped its architecture.

Recommendation Process Overview

Recommendation aims to match users with items they may like, following a loop of candidate generation, ranking, strategy, presentation, feedback, and evaluation.

Phase 1: Independent 1.0 (2011‑2013)

During this period the team was small and inexperienced, handling many parallel projects with a simple stack (Apache + mod_python, Redis, custom DBs). The focus was rapid implementation of business requirements, resulting in a lightweight but fragmented pipeline: candidate → strategy → simple display.

Advantages: simplicity, fast feature delivery, parallel project independence.

Drawbacks: incomplete workflow, lack of feedback/evaluation, poor algorithm support, difficult operations and testing.

Outcomes included the creation of the internal Woo framework, mapdb static storage, and a generic recommendation application framework.

Phase 2: Layered 2.0 (2013‑2014)

With a more mature team, the architecture shifted to a layered design covering the full recommendation loop: candidate, ranking, strategy, display, feedback, and evaluation. The stack incorporated Apache + mod_wsgi, Python for rapid development, C/C++ Woo services for high‑performance computation, and a mix of Redis, mapdb, and custom storage solutions.

Key components:

Application layer: common_recom_frame framework exposing project, work, and data interfaces.

Computation layer: extended Woo framework for efficient ranking.

Data layer: separate handling of static (Hadoop‑derived) and dynamic (Redis) data via R9‑interface and RIN pipelines.

Infrastructure services: monitoring, alerting, and offline evaluation UI.

Advantages: complete workflow, unified data handling, algorithm support, data‑first mindset, easier deployment and QA.

Shortcomings: not fully tailored to recommendation, limited algorithm training, reliance on developers for strategy implementation.

Results included the launch of core recommendation products (feed, user, content recommendations), open‑source release of lab_common_so, the Lushan static storage cluster, and the RUF framework.

Phase 3: Platform‑Centric 3.0 (2014‑present)

Focusing on effectiveness rather than pure feature expansion, the platform‑centric architecture abstracts common methods for candidate generation, ranking, training, and feedback, aligning the system more closely with algorithmic needs.

Key innovations:

Standardized all‑in‑one interface for the application layer.

Artemis and item‑cands modules for candidate generation.

EROS strategy platform for model training, feature selection, and online A/B testing.

Enhanced data pipelines (R9‑interface, RIN) supporting both static and dynamic candidates.

Benefits include deeper algorithm integration, streamlined candidate handling, and a unified input‑output contract.

Outcomes: migration of core recommendation products to the new platform, open‑source release of EROS training workflow, and standardized recommendation APIs.

Conclusion

The evolution of Weibo's recommendation architecture demonstrates a tight feedback loop between business demands and technical solutions, emphasizing incremental optimization, open‑source collaboration, and a data‑driven approach to improve recommendation quality.

Source: Weibo Recommendation Blog