Overview of Recommendation Systems and Their Evolution in Live Streaming Platforms

1. What Is a Recommendation System?

Recommendation systems help users discover content and alleviate information overload by modeling user interests from behavior and predicting items they may like.

Information Overload

When the amount of information exceeds a person's processing capacity, users cannot effectively integrate or internalize useful data, leading to reduced productivity and satisfaction.

Interest Modeling

Interest modeling infers future preferences based on past user interactions.

2. Early Recommendation Systems

Early Huajiao live‑stream recommendation relied on a simple hotness score calculated from viewer count, gifts, and interactions, which favored popular streams but suffered from head‑stream concentration and lack of personalization.

3. Modern Recommendation System Architecture

Modern pipelines consist of a coarse‑ranking (recall) stage that generates a candidate set from millions of items, followed by a fine‑ranking stage that refines the set using richer features and more complex models such as wide‑&‑deep or deep neural networks.

4. Recall vs. Ranking

Recall uses low‑cost, fast models (e.g., collaborative filtering) to filter candidates, while ranking applies comprehensive features and sophisticated models to order the final list.

5. Collaborative Filtering

5.1 Neighborhood‑Based Methods

Neighborhood modeling converts raw rating logs into a sparse matrix, then fills missing entries by computing cosine similarity between users or items.

Item‑based example: calculate item similarity matrix, then predict missing scores for a user.

Huajiao uses an item‑based model because the number of streamers is much smaller than users, making similarity matrices lightweight and interpretable.

5.2 Latent‑Factor Methods (Matrix Factorization)

Matrix factorization decomposes the rating matrix into two low‑dimensional latent matrices (X and Y) and can handle explicit or implicit feedback. Implicit feedback treats interaction intensity (e.g., watch time) as confidence weights.

Optimization is typically performed by alternating least squares (ALS).

5.3 Deep Learning‑Based Factorization

Neural Collaborative Filtering (NCF) replaces the inner‑product with a deep neural network to increase expressive power. NeuMF combines a generalized matrix factorization (GMF) layer with a deep network, learning both linear and non‑linear interactions.

6. Ranking Models

6.1 Feature Engineering

Features are derived from user profiles, streamer attributes, and recent interaction logs. Labels are generated by defining positive (e.g., watching >30 s) and negative samples.

6.2 Early Ranking Models

Logistic Regression (LR): simple, highly interpretable but limited to linear interactions.

Factorization Machines (FM): captures second‑order feature interactions.

GBDT+LR: tree‑based feature crossing followed by LR for automatic cross‑features.

6.3 Deep Ranking Models

Wide‑&‑Deep, DeepFM, DIN, and other architectures combine linear components (LR or FM) with deep neural networks to learn both memorization and generalization.

6.4 Multi‑Task Models

Models such as ESMM and MMOE jointly predict multiple objectives (click, watch time, gift, etc.) to achieve balanced optimization across business metrics.

7. Live‑Streaming Specific Considerations

Live streams are dynamic, requiring multimodal content understanding (text, images, video, audio) and real‑time features (viewer count, interaction rate) to capture hotness and user intent.

7.1 Model Structure for Huajiao

The pipeline includes raw input layers (user, context, item features), feature interaction layers (FM, CIN, Attention), feature concatenation, and multi‑task output towers.

7.2 Training and Deployment

Offline training uses daily aggregated samples stored in HDFS, while online incremental updates are performed via streaming (e.g., Flink). Models are served with TensorFlow Serving and accessed through high‑performance Go services.

8. Conclusion

The article summarizes the evolution of Huajiao’s recommendation system from simple hotness rules to sophisticated deep and multi‑task models, emphasizing that the best model is the one most suited to the specific live‑streaming scenario rather than the most complex.