Integrating Flink with TensorFlow for End-to-End Machine Learning Pipelines

Deep learning is increasingly applied in recommendation, search, face recognition, translation, autonomous driving and many other domains, and its integration with data‑processing frameworks such as Flink is essential for end‑to‑end machine‑learning solutions.

A typical ML workflow includes feature engineering, model training, and offline or online prediction, each generating logs that must be processed before the next step.

Using separate Flink and TensorFlow engines makes deployment complex, requires manual IP/port configuration for distributed TensorFlow, and lacks automatic failover.

The proposed approach runs TensorFlow programs directly on a Flink cluster, allowing a single computation engine to handle feature engineering, model training, and real‑time prediction, simplifying deployment and saving resources.

Flink abstracts all computations as operators (sources, sinks, and various processing operators), forming a directed topology that can ingest and emit data streams.

In a machine‑learning cluster, nodes are grouped as workers (performing computation) and parameter servers (updating parameters); these groups map to Flink operators.

The Flink‑ai‑extended project introduces two abstraction layers: an ML Framework that connects Flink with external ML engines, and an ML Operator that provides interfaces to add an Application Manager role (am) and additional node roles.

The ML Framework defines two roles: an Application Manager that manages node lifecycles, and node roles that execute ML algorithms.

The ML Operator offers addAMRole to insert an Application Manager into a Flink job and addRole to add groups of ML nodes, enabling a cluster composed of an AM and multiple node groups.

Flink operators run as Java processes, while TensorFlow nodes typically run as Python processes; data exchange between them is achieved via shared memory.

TensorFlow distributed training uses worker and parameter‑server (ps) roles. In batch mode, source operators launch the appropriate number of workers and ps instances, and communication follows TensorFlow’s native gRPC protocol.

In streaming mode, data is first processed by source and join operators, then fed to TensorFlow workers via UDTF or flatMap operators, with parallelism matching the number of worker nodes.

For inference, a Python‑based pipeline loads the trained model into ps nodes and processes incoming data through workers, writing prediction scores back to Flink.

A Java‑only inference path can load a TensorFlow SavedModel directly via the TensorFlow Java API, eliminating the need for Python processes and ps nodes, and allowing predictions to be emitted to downstream Flink operators.

In summary, the article details the principles of Flink‑ai‑extended and demonstrates how Flink combined with TensorFlow can support both model training and real‑time prediction within a unified, resource‑efficient workflow.