Inside Weibo’s Recommendation Engine: Architecture, Layers, and Key Technologies

Recommendation System Overview

Weibo’s recommendation system is organized into four layers: the front‑end presentation layer (RFront), the application layer (RApp), the compute layer (RCompute), and the data layer (RStore). The data layer also includes logging, monitoring, and evaluation components.

1. Recommendation Front‑End (RFront)

RFront is responsible for rendering Weibo content on web and client applications. Although the presentation differs across platforms, the underlying data and methods are shared, requiring a maintainable and adaptable layer that can quickly respond to product needs.

2. Recommendation Application (RApp)

RApp provides candidate items and performs part of the ranking for the front‑end. It offers APIs built on a common framework (CRF) that originally used Apache + mod_python and later migrated to nginx + Lua. The framework unifies data access across protocols (memcached, Redis, OpenAPI) and defines a project‑based development model.

Key Features of CRF

Transparent data retrieval across different storage protocols.

Project abstraction allowing developers to extend functionality while exposing core work interfaces.

Standardized recommendation interfaces (e.g., obj, tobj, from, appkey, num, type, pid).

3. Recommendation Compute Layer (RCompute)

RCompute handles CPU‑intensive tasks such as second‑order relationship calculations, bridge‑user analysis, and CTR estimation. It uses the Lab_common_so framework, derived from the Woo framework, which integrates data resources, standardizes workflows, and supports local data caching.

4. Data Layer (RStore)

RStore stores diverse data types, including static mined data, real‑time user behavior, attribute caches, and offline computation results. Different storage solutions are chosen based on data characteristics: Redis/Memcached for online data, Lushan/MapDB for offline static data, and custom formats for high‑volume feature compression.

The layer follows an IN/OUT/STORE model: IN unifies data entry via RIN, OUT provides transparent data access for downstream services, and STORE manages heterogeneous storage back‑ends.

5. Supporting Tools: Logging, Monitoring, Evaluation, and Alerting

Logging systems collect both offline analysis data and online monitoring data, integrating logs from internal systems and the Hadoop cluster. Monitoring is divided into performance monitoring, effect monitoring, and A/B test comparison, with dashboards illustrating exposure, performance, and effectiveness metrics.

Evaluation comprises demo experiments, offline evaluation, and online evaluation, while alerting mechanisms are being refined to meet customized performance and effect thresholds.

The article concludes that the overview of Weibo’s recommendation architecture will be expanded in future detailed posts.