SQLFlow: End‑to‑End AI Workflow Construction Using SQL

End‑to‑end machine learning aims to hide complex technical details from business users and provide automatic model adjustment; SQLFlow, released by Ant Group in 2019, is an open‑source compiler that builds AI workflows using only SQL.

SQLFlow connects various database systems (MySQL, Hive, MaxCompute) and machine‑learning engines (TensorFlow, Keras, XGBoost), compiling SQL programs into distributed workflows that cover data extraction, preprocessing, model training, evaluation, prediction, explanation, and even operations planning.

Using SQL to describe AI tasks reduces the barrier to building AI applications because SQL expresses intent (what to do) rather than process (how to do it), allowing concise statements that the system expands into full pipelines.

Example – training a DNN classifier on the Iris dataset: SELECT * FROM iris.train TO TRAIN DNNClassifier WITH model.n_classes = 3, model.hidden_units = [10, 20] LABEL class INTO my_dnn_model;

Model evaluation: SELECT * FROM iris.test TO EVALUATE my_dnn_model WITH validation.metrics="Accuracy" LABEL class INTO iris.evaluate_result;

Prediction: SELECT * FROM iris.pred TO PREDICT iris.pred_result.class USING my_dnn_model;

Model explanation (SHAP/TensorFlow integration): SELECT * FROM iris.test TO EXPLAIN my_dnn_model;

SQLFlow supports a rich set of built‑in models, including DNN, RNN, LSTM, XGBoost trees, Deep Embedding Clustering, k‑means, scoring‑card models, ARIMA, STL time‑series models, and more.

Beyond single tasks, a full SQL program can be compiled into an Argo workflow that runs on a Kubernetes cluster; the system automatically analyses dependencies, maximises concurrency, and maps each step to the appropriate engine.

The Model Zoo framework lets developers contribute reusable models to a private or public repository, enabling business users to reference them directly in SQL without writing code.

Real‑world case studies include:

Fund inflow/outflow forecasting using SELECT time, purchase FROM fund.train TO TRAIN sqlflow_models.ARIMAWithSTLDecomposition WITH model.order=[7,0,2], model.period=[7,30], model.date_format="%[2]s", model.forecast_start='2014-09-01', model.forecast_end='2014-09-30' LABEL purchase INTO purchase_predict_model;

Click‑through‑rate prediction with a Deep & Wide model, using column‑wise feature specifications and embedding of categorical features (full SQL shown in source).

Driver shift‑preference clustering using SELECT * FROM driver_log.train TO TRAIN sqlflow_models.DeepEmbeddingClusterModel WITH model.n_clusters=5 INTO cluster_driver_model; followed by prediction and visualization.

In summary, SQLFlow bridges databases and AI systems, automatically compiles SQL programs into scalable, concurrent workflows on Kubernetes, provides extensive built‑in models and a Model Zoo, and enables engineers to focus on modeling while dramatically lowering the cost and time of building AI solutions.