Inside Toutiao’s Recommendation Engine: Architecture, Features, and Evaluation

System Overview

The recommendation function is formalized as y = F(X_i, X_u, X_c), where X_i are content features, X_u are user features, and X_c are contextual features (location, time, device). The platform supports a mixture of algorithms—collaborative filtering, logistic regression, factorization machines, GBDT, and deep neural networks (often LR+ DNN or LR+ GBDT)—and provides an extensible experimentation framework for rapid model iteration.

Feature Engineering

Relevance features : keyword, category, source, topic matches; implicit similarity from vector distances.

Environmental features : geographic location, time of day, device type.

Popularity features : global, category, and topic hotness, useful for cold‑start.

Collaborative features : user‑behavior similarity (click, interest, topic) to mitigate recommendation narrowing.

Model Training and Deployment

Training runs in near‑real time on a Storm cluster that consumes user actions (click, impression, like, share) from Kafka. A custom high‑performance parameter server stores billions of raw and vector features. The pipeline records features, builds training samples, and updates model parameters continuously; latency is dominated by user feedback delay.

Because the content pool is massive, a two‑stage retrieval is used: an offline inverted index (keyed by category, topic, entity, source) provides candidate pools, and an online recall module selects a few thousand candidates within a 50 ms budget, ranking them by hotness, freshness, and recent user actions.

User Tag Generation

User tags include interests, topics, keywords, source preferences, vertical attributes (e.g., car model, sports team), demographics (gender, age), and location. Initially batch‑computed on Hadoop, the system migrated to a Storm‑based streaming pipeline that updates tags in near real time for tens of millions of users, reducing CPU usage by ~80%.

Evaluation Framework

An A/B testing platform partitions users offline, assigns traffic to control and variant groups, and collects metrics (CTR, dwell time, likes, shares, etc.) in near real time. Multiple metrics are evaluated simultaneously to avoid over‑optimizing short‑term signals and to preserve long‑term health. Experiments are automatically allocated and reclaimed.

Content Safety

Content passes a multi‑layer safety pipeline. Professionally generated content (PGC) undergoes risk review before wide distribution. User‑generated content (UGC) is first filtered by a risk model; flagged items receive secondary manual review. Models for pornography, profanity, and low‑quality content are trained on large multimodal datasets with a recall target ≥95 % (precision is sacrificed when necessary). Human reviewers continuously adjust thresholds and handle edge cases such as fake news and click‑bait.

Key Engineering Challenges

Real‑time training on billions of features requires a custom parameter server optimized for the scale of Toutiao.

Recall must be extremely fast; the offline inverted index and online filtering together keep end‑to‑end latency under 50 ms.

Cold‑start for new articles is mitigated by popularity and semantic features (topic, keyword embeddings).

User tag computation moved from batch Hadoop jobs to a Storm streaming system to achieve near‑real‑time updates with 80 % CPU savings.