Deepthought: An End‑to‑End Machine Learning Platform at iQIYI

Deepthought is a one‑stop machine learning platform developed by iQIYI to address the pain points of traditional single‑machine script‑based development, such as low readability, high coupling, difficulty of reuse, and limited scalability for large‑scale data and models.

The platform was designed with four core business requirements: (1) encapsulating distributed machine‑learning frameworks with a focus on open‑source solutions; (2) decoupling each stage of the ML pipeline to enable reusable outputs; (3) deep integration with the big‑data workflow system Tongtian Tower for both online and offline task execution; and (4) reducing user code pressure through visual, drag‑and‑drop configuration.

Deepthought has evolved through three major versions:

v1.0 – a fraud‑detection‑oriented platform that wrapped Spark ML/MLLib binary classification models and basic preprocessing (missing‑value filling, normalization). It emphasized feature management and data configuration.

v2.0 – a generic platform that introduced component‑based management and scheduling, a richer set of algorithms (supervised classification, regression, clustering, graph algorithms), automatic hyper‑parameter tuning, and visual interaction for building pipelines. It also added offline timed‑task scheduling via Tongtian Tower.

v3.0 – added real‑time inference services supporting HTTP and RPC, automatic hyper‑parameter optimization (random search, grid search, Bayesian optimization, evolutionary algorithms), and a parameter‑server implementation to train models on ultra‑large datasets (millions of dimensions, billions of rows). Real‑time services are deployed through QAE, Skywalker/Dubbo, and monitored by Hubble/Venus, using a thread‑pool + pipeline architecture for low‑latency predictions.

Key technical implementations include:

Spark ML/MLLib wrappers with additional algorithms such as GBDT encoding, which stores tree details and performs parallel data encoding.

Data‑preprocessing components (stratified sampling, compression, encoding) and automatic label mapping.

Real‑time loss curve output built on Spark’s messaging and event‑bus mechanisms, providing TensorFlow‑like loss visualization.

Hot‑load architecture for real‑time services that allows model updates without killing QAE instances, supporting multiple models per instance and explicit version selection.

Deepthought is currently used in several iQIYI teams, including traffic anti‑fraud, user‑behavior analysis, recommendation, and literary content. In anti‑fraud, it leverages clustering, isolation forest, and binary classifiers (LR, GBDT, XGBoost) with automated hyper‑parameter tuning, achieving >97% accuracy and recall. In recommendation, it provides GBDT encoding, SQL‑customizable components, collaborative filtering, and FM models, enabling end‑to‑end training, evaluation, and real‑time deployment.

The platform’s future roadmap focuses on improving usability and stability, expanding real‑time inference support, enriching model types (including deep learning), and further streamlining the end‑to‑end ML workflow.