Inside Toutiao’s Real‑Time Recommendation Engine: Architecture, Features, and Evaluation

System Overview

Recommendation is modeled as fitting a function y = F(X_i, X_u, X_c) where X_i are content features, X_u user features, and X_c environment features. Content includes text, images, video, and UGC; user features cover interests, demographics, and implicit interests; environment captures location, time, and scenario.

Algorithm Implementation

Various models can be applied, such as collaborative filtering, logistic regression, deep learning, factorization machines, and GBDT. An industrial‑grade system must support a flexible experiment platform to combine and adjust model structures, as no single architecture fits all scenarios.

The main feature categories are:

Relevance features – explicit and implicit matching between content and user.

Environment features – geographic and temporal context.

Popularity features – global, category, and keyword hotness.

Collaborative features – behavior similarity to mitigate filter‑bubble effects.

Real‑time training runs on a Storm cluster, processing click, impression, share, and other actions. A custom high‑performance parameter server stores model weights; data flow is user actions → Kafka → Storm → label generation → online model update, achieving near‑real‑time latency.

Recall Strategies

Because the content pool contains billions of items, a fast recall stage selects a few thousand candidates per request. An offline inverted index keyed by category, topic, entity, or source is used; online recall quickly truncates this list based on user interest tags, freshness, and popularity, keeping latency under 50 ms.

Content Analysis

Text analysis provides explicit semantic tags and implicit topics/keywords, which are crucial for building user interest models and solving cold‑start problems. Similarity features help avoid duplicate recommendations, while spatial‑temporal and quality signals (spam, low‑quality) further refine ranking.

User Tagging

User tags include interest categories, topics, keywords, source, clustering, and vertical interests (e.g., car models, sports teams, stocks), as well as demographics such as gender, age, and location. Tags are computed in batch on Hadoop and, since 2014, in a real‑time Storm pipeline for millions of users, reducing CPU usage by about 80%.

Evaluation

Evaluation combines multiple metrics (click‑through rate, dwell time, likes, shares) and long‑term ecosystem health. A/B testing uses offline bucketing and online traffic allocation; data are collected hourly but analyzed daily. An automated experiment platform assigns traffic, computes confidence intervals, and generates conclusions and optimization suggestions.

Content Safety

Content safety relies on manual review and AI models for porn, profanity, low‑quality, and misinformation detection. PGC content undergoes risk review; UGC passes a risk model before a second‑stage review if flagged. High‑recall models (≥95%) are used, with human verification for final decisions.