Galileo: A Distributed Graph Deep Learning Framework for Large‑Scale Industrial Scenarios

Introduction Graphs are an effective way to represent relational data in retail scenarios, where people, goods, and places are naturally linked. JD Retail’s Galileo platform provides offline/online graph computation and graph database services, and this article focuses on its graph deep‑learning framework.

Graph Neural Networks (GNN) GNNs are deep‑learning methods for processing graph‑structured data, addressing relational reasoning and interpretability. They are applied in recommendation, risk control, knowledge graphs, computer vision, and NLP.

Academic and Industrial Landscape Recent years have seen rapid growth in graph deep‑learning research at top conferences, with open‑source libraries such as DGL and PyTorch Geometric. Industry contributions include Alibaba’s Euler, Tencent’s Plato, and Baidu’s Paddle Graph Learning.

Core Challenges The main challenges for GNN deployment are heterogeneous graph handling, dynamic graph learning, training on industrial‑scale data, and framework usability.

Architecture Design Galileo is a distributed graph deep‑learning framework featuring five key advantages: support for heterogeneous and dynamic graphs, ultra‑large‑scale training, custom model definition, and ease of use. It adopts a layered design consisting of a Graph Engine, Deep Learning Backend, and Graph Algorithm Layer.

Graph Engine Provides data loading, sampling, and query services, optimized for billions of vertices and hundreds of billions of edges. It supports heterogeneous graphs, both static and dynamic modes, memory‑efficient storage (reducing memory usage by ~70%), and fast binary data loading (improving load speed by ~40%). Rich sampling operators include random walk, weighted node/edge sampling, neighbor sampling, attribute queries, timestamp‑based bias sampling, and subgraph sampling.

Deep Learning Backend Seamlessly integrates TensorFlow and PyTorch, offering Dataset abstractions (NodeDataset, EdgeDataset, FileDataset) and a Transform interface for custom preprocessing. Multiple training modes are supported: single‑CPU/GPU, multi‑GPU, and distributed training via a self‑developed Parameter Server for ultra‑large models.

Graph Algorithm Models Galileo ships built‑in models such as DeepWalk, Node2vec, LINE, Metapath2vec, GraphSAGE, and GAT, with implementations matching original papers. It also provides common graph layers (graph convolution, attention, neighbor aggregation) and a custom model interface for researchers to implement new algorithms.

Productization and Performance The C++‑based distributed graph engine achieves high‑throughput subgraph partitioning and parallel execution. The platform supports both TensorFlow and PyTorch, with plans to add more frameworks, and emphasizes consistency, extensibility, and ease of use.

Usability and Ecosystem Galileo defines a unified text‑based graph data format and offers efficient distributed conversion tools. It is integrated into JD Retail’s Easy‑Algorithm platform, delivering end‑to‑end services from data generation to model training and analysis.

Future Outlook Galileo is already serving internal scenarios such as search, recommendation, knowledge graph, and anti‑fraud, with notable performance gains. The team plans further optimizations, offline‑online integration, and open‑source release by year‑end.

Team Introduction The Galileo project belongs to JD Retail’s Data Algorithm Channel, comprising experienced architects and engineers from leading internet companies and top universities, focusing on industrial‑grade data algorithms and large‑scale machine learning.