How Alibaba’s Alink Empowers Real‑Time Machine Learning on Flink

Background

With the rise of big data and AI, machine‑learning models must handle massive batch data and provide real‑time predictions on streaming data, often within enterprise micro‑services. Distributed clusters and online learning have become essential for timely market responses.

What Is Alink?

Alink is a next‑generation machine‑learning algorithm platform developed by Alibaba’s PAI team since 2017, built on the real‑time compute engine Flink. It offers a comprehensive algorithm component library and a convenient framework that lets developers build end‑to‑end pipelines covering data processing, feature engineering, model training, and prediction.

Leveraging Flink’s batch‑stream integration, Alink provides consistent operations for both batch and streaming tasks. It was first deployed internally in 2018 and later open‑sourced.

Relation to FlinkML

FlinkML is the existing Flink machine‑learning library, which is limited in algorithms and data structures. Alink is a completely new implementation based on the latest Flink, with no code overlap with FlinkML. Alibaba contributes Alink improvements back to FlinkML, and many Alink concepts (e.g., Params) have already been adopted.

Why Alink Over SparkML?

Alink benefits from Flink’s engine, includes extensive optimizations (communication, data access, iterative processing), and provides online‑learning algorithms that excel in high‑frequency update scenarios such as news feeds. While offline capabilities are comparable to SparkML, Alink’s unique streaming and online‑learning support give it a clear advantage.

Main Features and Advantages

Rich and Efficient Algorithm Library

Alink offers a wide range of batch and streaming algorithms, all accessible via a Python interface, enabling developers to complete the full ML workflow from data handling to prediction.

User‑Friendly Experience

PyAlink provides a notebook‑based interactive environment, supporting both local and cluster execution, and integrates Alink operators with DataFrames for seamless Python workflows.

Efficient Iterative Computation Framework

Alink includes an intermediate function library featuring the Iterative Communication/Computation Queue (ICQ), which dramatically improves performance over Flink’s native iterative APIs.

DataSet<Row> model = new IterativeComQueue()
    .initWithPartitionedData(OptimVariable.trainData, trainData)
    .initWithBroadcastData(OptimVariable.model, coefVec)
    .initWithBroadcastData(OptimVariable.objFunc, objFuncSet)
    .add(new PreallocateCoefficient(OptimVariable.currentCoef))
    .add(new PreallocateCoefficient(OptimVariable.minCoef))
    .add(new PreallocateLossCurve(OptimVariable.lossCurve, maxIter))
    .add(new PreallocateVector(OptimVariable.dir, new double[]{0.0, OptimVariable.learningRate}))
    .add(new PreallocateVector(OptimVariable.grad))
    .add(new PreallocateSkyk(OptimVariable.numCorrections))
    .add(new CalcGradient())
    .add(new AllReduce(OptimVariable.gradAllReduce))
    .add(new CalDirection(OptimVariable.numCorrections))
    .add(new CalcLosses(OptimMethod.LBFGS, numSearchStep))
    .add(new AllReduce(OptimVariable.lossAllReduce))
    .add(new UpdateModel(params, OptimVariable.grad, OptimMethod.LBFGS, numSearchStep))
    .setCompareCriterionOfNode0(new IterTermination())
    .closeWith(new OutputModel())
    .setMaxIter(maxIter)
    .exec();

Case Studies

Sentiment Analysis

A pipeline with missing‑value filling, Chinese tokenization, stop‑word filtering, vectorization, and logistic regression is used to analyze hotel review data.

FTRL Online Learning

In online advertising, the FTRL algorithm is employed for real‑time click‑through‑rate prediction, with a pipeline that includes standardization and feature hashing.

Future Plans

Alink will continue collaborating with the community to improve functionality, performance, and usability, and aims to merge fully into FlinkML, eventually matching SparkML’s capabilities.