GPU-Accelerated Model Training Optimizations for Snowball Feed Recommendation System

Snowball’s headline feed uses a typical Feeds architecture and a Wide&Deep model built on TensorFlow to rank content for personalized recommendation. The rapid changes in the stock market demand timely and efficient distribution of user‑generated content.

The system consists of three layers—online request handling, offline data processing, and a base data layer. Offline model training becomes a bottleneck: daily logs generate about 5 TB of data and 10 billion samples, with peak online throughput of 400 QPS and latency under 600 ms, requiring faster model iteration.

To address the compute‑intensive workload, the team selected GPU acceleration. CPUs handle data loading, feature extraction, and shuffling, while GPUs perform the heavy matrix and floating‑point calculations, offering 30‑100× speedup over CPUs for such operations.

The initial GPU implementation performed worse than the CPU baseline (6 k vs 8 k samples/s) and suffered low utilization (CPU <20%, GPU 0‑8%).

Several optimizations were applied:

Multi‑threaded data preparation increased training speed to 8 k samples/s, yielding about a 30% improvement over pure CPU training.

Continuous data pre‑preparation kept the GPU fed while the CPU prepared the next batch, raising speed to 9 k samples/s (~10% gain).

Introducing the TFRecord format eliminated repeated feature extraction, boosting speed to 1.6 w samples/s (10% gain) but raised disk I/O to 90% utilization.

Compressing TFRecord files—through multi‑sample merging and GZip—reduced storage by 56%, lowered disk usage, increased GPU utilization to ~20%, and lifted speed to 2.1 w samples/s (30% gain).

Reordering batch and map operations (batch‑map) further improved throughput to 2 w samples/s.

A summary table lists each optimization, the resulting training speed (samples/second), GPU utilization, and remarks.

The optimized pipeline was integrated into the Model_bus system, which automates feature conversion, model training, evaluation, and serving. Feature conversion pulls raw data from HDFS, extracts features, writes compressed TFRecord files back to HDFS, and notifies the training service via MQ. The training service consumes TFRecord data, trains and evaluates models (calculating AUC), and uploads successful models to HDFS, triggering the serving service built on TensorFlow‑Serving. Model status and metrics are logged to MySQL and displayed via a web UI.

With these GPU‑accelerated improvements, daily model updates are completed within five hours, meeting production requirements. Future work includes building a real‑time recommendation system, a real‑time data warehouse, and distributed TensorFlow training to further enhance computational capacity.

The article concludes with a recruitment notice seeking recommendation system or algorithm engineers, providing a contact email for interested candidates.