Baidu Short Video Push System: Architecture Design and Billion-Level Data Optimization Practice

Introduction: The Short Video Push System is a distributed Push system supporting multiple Baidu apps and business scenarios. Currently, it supports push services for apps like Haokan Video, Live Streaming, Du Xiaoshi, and Haokan Big Font Version. It provides personalized push based on user characteristics, operational push for hot activities and events, and real-time push based on follower or subscription relationships. The goal is to efficiently deliver personalized content to users through notification bar messages to improve user engagement and retention.

System Overview: The Push system serves multiple business units with hundreds of millions of quarterly active users. The system subscribes to video material information and user attribute updates in real-time, recalls personalized materials at 1 AM daily, creates push tasks based on scheduled times, performs preprocessing half an hour before sending, and delivers messages through the Cloud Push platform to device vendors.

Core Architecture Modules: 1) Material Center stores video metadata including titles, descriptions, and images. 2) User Center stores user basic info and Push-specific attributes like estimated active time windows. 3) Personalized Recall performs material recall for quarterly active users. 4) Realtime-API handles real-time push scenarios. 5) Frequency Control (UFC) prevents user disturbance with daily and hourly limits. 6) Preprocessing Service handles task splitting and message construction. 7) Sending Service delivers tasks to vendors based on priority. 8) Receipt Service processes delivery confirmations. 9) Control Center provides visual configuration.

Optimization 1 - Scheduled Push Time Estimation: Originally, all users received their first personalized push at 6:30 AM and last at 9:45 PM. The system now estimates individual user activity patterns by analyzing their first active time in 7-day windows across different time periods, dynamically adjusting push timing to improve click-through rates.

Optimization 2 - User Segmentation Service: The original architecture used physical machines and Hadoop clusters with FTP/AFS file loading (30s per task). The new architecture uses bitmap-based user groups with logical expressions, reducing loading time and enabling efficient set operations through Redis bitmap storage.

Optimization 3 - Frequency Control Service: Replaced fixed hash-based distribution with consistent hashing for dynamic scaling. Implemented Protobuf protocol for data compression, achieving 2x faster serialization than JSON and 75% reduction in Redis resource usage.