DLFlow: An End-to-End Deep Learning Solution for Big Data Offline Tasks

DLFlow is a deep learning solution developed by the user profile team at Didi, designed for big data offline task environments. By combining Spark and TensorFlow, it enables rapid processing of original features, large-scale distributed training, and massive distributed prediction, significantly improving model development efficiency.

The framework addresses the deployment challenges of deep learning applications in offline production tasks. While popular deep learning frameworks focus on online service deployment, offline environments have been largely overlooked. DLFlow explores combining GPU and Spark clusters to handle large-scale data processing in offline settings.

DLFlow provides an end-to-end solution covering feature preprocessing, model building, training, and deployment. It abstracts the deep learning pipeline into tasks and models, supports configuration-driven execution, and includes automated feature processing, pipeline automation, and best practices for offline production scenarios.

The framework consists of four layers: core capabilities (task scheduling, core modules, model tasks), and underlying support (TensorFlow, Spark, Hadoop). It uses a workflow engine to manage dependencies between tasks and configurations, enabling easy pipeline construction.

DLFlow is available via pip installation and supports custom model development through inheritance from ModelBase. Users can configure the system using HOCON files and run tasks with simple commands. The framework includes built-in models and tasks, with plans to expand the model library.

DLFlow significantly improves model development efficiency by allowing developers to focus on model design while the framework handles preprocessing, pipeline management, and deployment tasks.