Inside ByteDance’s Recommendation Engine: How TikTok Delivers Billions of Personalized Feeds

System Overview

The recommendation problem is formalized as fitting a function y = F(X_content, X_user, X_context) that predicts a user's satisfaction with a piece of content. The three groups of input variables are:

Content features : extracted from heterogeneous media (articles, images, short videos, Q&A, micro‑posts) using dedicated pipelines.

User features : explicit interests, demographics (age, gender, occupation) and implicit interests derived from behavior models.

Contextual features : mobile‑centric signals such as location, time of day and device state.

The model outputs a relevance score for each user‑content pair in a given context.

Modeling Approaches and Architecture

The supervised learning task can be solved with a range of algorithms, including:

Collaborative‑filtering and logistic regression.

Factorization Machines.

Gradient‑Boosted Decision Trees (GBDT).

Deep neural networks (DNN) and hybrid LR‑DNN or LR‑GBDT architectures.

ByteDance operates a flexible experimentation platform that allows mixing model components and adjusting architectures per product line.

Real‑time training is used for most recommendation products. The pipeline streams raw events to Kafka, processes them with a Storm cluster to construct labeled samples, and updates model parameters online.

Feature Engineering and Recall

Four major feature categories drive recommendation decisions:

Relevance features : keyword, category, source, topic matches and implicit similarity from vector distances.

Environmental features : geographic location and temporal context.

Popularity (hotness) features : global, category and keyword hotness, useful for cold‑start.

Collaborative features : similarity of user behavior patterns (click, interest, topic, word‑vector similarity) to broaden exploration.

Recall is performed via an offline inverted index built on keys such as category, topic, entity and source. The online stage truncates candidates based on user interest tags, targeting sub‑50 ms latency and selecting a few thousand items from billions of candidates.

Content Analysis

Text analysis is the cornerstone for user interest modeling. Two types of semantic tags are extracted:

Explicit semantic tags : manually defined taxonomy (e.g., “technology”, “sports”).

Implicit semantic features : topics and keywords derived from statistical models (e.g., LDA, word embeddings).

Entity recognition combines tokenization, part‑of‑speech tagging and knowledge‑base lookup to resolve multi‑word entities. Similarity detection mitigates duplicate recommendations by comparing article topics, writing style and entities, while allowing user‑specific tolerance (e.g., sports fans may prefer repeated coverage).

User Tagging

User tags include interest categories, topics, keywords, source preferences, clustered interests (e.g., car models, sports teams, stocks) and demographic attributes (gender, age, location). Demographic data are derived from third‑party social logins or inferred from device usage patterns.

Tag generation evolved from a daily Hadoop batch job (processing two months of activity for millions of users) to a Storm‑based streaming system launched in 2014. The streaming system updates tags in near real‑time for high‑frequency actions while retaining daily updates for static attributes.

Evaluation and Experimentation

Evaluation uses a comprehensive metric suite beyond click‑through rate, including dwell time, likes, comments, shares and downstream business impact. A/B testing is orchestrated by an internal platform that pre‑assigns users to buckets, distributes traffic, and automatically generates statistical reports, confidence intervals and optimization suggestions.

Typical experiments allocate 10 % of traffic, split evenly between baseline and variant groups. Results are observed hourly but aggregated daily to smooth variance.

Content Safety

Content originates from professional (PGC) sources and user‑generated (UGC) sources. UGC passes a risk model before manual review; high‑risk items trigger secondary review and possible takedown.

Safety models include porn detection, profanity detection and low‑quality content detection, trained on massive multimodal datasets. The system prioritizes high recall (acceptable precision) to filter harmful material before it reaches users.