Distributed Machine Learning Framework GDBT for High‑Dimensional Real‑Time Recommendation Systems

With the rapid growth of internet data and increasingly complex AI models, recommendation systems now face challenges of massive data volume, high‑dimensional features, and strict real‑time constraints. The talk outlines how distributed machine learning frameworks can address these issues.

Challenges for recommendation systems include handling massive data and high‑dimensional features to improve effectiveness, providing hard‑real‑time (millisecond‑level) and soft‑real‑time (hour‑to‑day level) features for timely user interest capture, and fully exploiting data value, all of which demand abundant AI infrastructure compute power.

Performance bottlenecks in large‑scale distributed ML arise from exponential data growth (Moore's law slowdown), high model dimensionality exceeding single‑machine memory, and high model‑update frequency consuming huge compute resources.

Solution approaches :

Use heterogeneous computing (GPU, accelerator cards) and distributed storage to scale compute.

Employ a large‑scale parameter server to store and update model parameters, applying NUMA‑friendly data structures, RWSpinLock for thread safety, and RDMA for low‑latency, high‑bandwidth communication.

Adopt streaming computation to improve model timeliness while reducing compute cost.

GDBT framework architecture consists of three core components:

Distributed DataSource (data parallelism) with load‑balancing work‑stealing and support for multiple data formats.

Parameter Server acting as a distributed in‑memory database, slicing parameters across nodes and using RDMA to accelerate gradient pull/push.

Workloads such as distributed SGD and tree‑model training, where GDBT supports both ranking‑based and histogram‑based tree learning, leveraging GPU/FPGA acceleration and efficient all‑reduce operations.

Experimental results show that the custom pDataSource achieves 30% memory usage while delivering 120% of Spark 2.4.4 performance, and the parameter server can handle billions of KV updates per second.

Network pressure and optimization are addressed by adopting RDMA, which provides sub‑microsecond latency, >100 Gb/s bandwidth, kernel bypass, and remote memory access, dramatically reducing the network bottleneck of model synchronization.

Finally, the online inference pipeline loads offline‑trained models from HDFS into the parameter server, serves scoring requests, and updates models via streaming pipelines, achieving near‑second model refresh cycles despite the complexity.

The presentation concludes with a summary of the engineering solutions that enable high‑performance, real‑time recommendation at scale.