Building an AI Ecosystem with Flink: Overview of AI Flow and Its Architecture

Background: Flink in the AI ecosystem In modern big‑data scenarios, Flink + AI solutions are increasingly common. AI Flow provides a top‑level workflow abstraction for end‑to‑end machine‑learning lifecycle management, and this article focuses on Flink’s role within AI Flow.

1. Lambda Architecture The classic Lambda architecture combines a batch layer and a speed (stream) layer to balance cost and real‑time requirements, but maintaining separate code bases is costly. Flink’s unified stream‑batch model allows a single code base to implement Lambda architecture.

2. AI task processing pipeline AI tasks typically consist of three stages: data preprocessing, model training, and inference, each with real‑time demands. The preprocessing stage involves feature engineering and sample stitching, which benefits from a unified batch‑stream engine. Training must handle data distribution drift and support online updates, while inference requires low latency.

3. Why choose Flink? Flink offers a unified engine for both online and offline preprocessing, supports running TensorFlow and PyTorch models, and thus serves as an ideal backbone for AI Flow.

AI Flow Overview

1. Why AI Flow? AI Flow manages the lifecycle of machine‑learning pipelines, providing a top‑level abstraction for AI workflows.

Library for managing ML pipeline lifecycle.

Top‑level abstraction for AI + big‑data workflow.

2. AI Graph AI Flow represents workflows as a directed acyclic graph (AI Graph) composed of AI Nodes (data ingestion, processing, training, prediction, evaluation) and AI Edges (Data Edge for data dependencies, Control Edge for scheduling dependencies).

3. AI Flow Working Principle AI Flow consists of two modules: the SDK for defining and compiling workflows, and the Service for executing them on Local, Kubernetes, or YARN. The SDK translates AI Graphs into jobs (Python, Flink, Spark) via a pluggable Job Generator. The Service includes Metadata Service, Model Center, and Notification Service.

① Metadata Service Manages metadata such as projects, datasets, workflow jobs, models, and artifacts, enabling monitoring and management of experiments and jobs.

② Model Center Handles model visualization, versioning, parameter management, status tracking, and lifecycle management.

③ Notification Service Supports workflow scheduling by notifying dependent jobs when a key is updated, e.g., when a new model is produced, downstream jobs receive notifications for evaluation or online prediction.

4. Value of AI Flow

Supports online scenarios.

Engine‑agnostic: works with Python, Flink, Spark.

Deployable on Local, Kubernetes, YARN.

Provides a top‑level abstraction for AI workflows.

Flink AI Flow implements AI Flow using Flink as the execution engine, leveraging Flink’s AI ecosystem (Flink ML Pipeline, Alink, PyFlink, TensorFlow/PyTorch on Flink).

1. Flink AI Flow Architecture

Flink AI Flow adds extensive data source support compared to the generic AI Flow architecture.

2. Flink ML Pipeline & Alink

The pipeline abstracts Transformers (data processing) and Estimators (model training).

Alink rewrites most machine‑learning libraries for Flink ML Pipeline.

3. Relationship between Flink AI Flow and ML Pipeline

Each box represents an ML Pipeline composed of AI Nodes; the Flink Job Generator translates these into Flink jobs, forming a DAG.

4. AI Flow and Python While many AI scenarios are Python‑based, AI Flow can run Python jobs directly or embed PyFlink to leverage Flink’s ecosystem.

5. TensorFlow on Flink TF on Flink enables TensorFlow code to run as a Flink operator, supporting online training with Flink’s real‑time capabilities.

Flink AI Flow Application Example: Advertising Recommendation

Real‑time user behavior data is fed into an online training module, producing dynamic model versions hourly. The Model Center manages these models, while the Notification Service triggers evaluation, validation, and online prediction modules to ensure accurate ad delivery.

Thank you for reading.