Understanding Flink Cluster Startup and Job Execution Process

Preface

This article is divided into two parts: how to start a Flink cluster and how a Flink job runs on the cluster (stream mode). It focuses on the internal working principles rather than implementation details.

Flink Cluster Startup

Flink deployment consists of JobManager and TaskManager. JobManager handles job distribution and slot resource management, while TaskManager runs the tasks.

The classic architecture diagram (although a bit outdated) still reflects most of the functionality.

Flink code is compiled on the client side before being submitted to JobManager, which is important for optimizer work.

JobManager receives the compiled job and distributes it to TaskManager slots.

Flink uses two communication mechanisms: Actor for remote calls and Netty for TaskManager data transfer.

JobManager integrates Scheduler, Checkpoint Coordinator, and other core components.

Ways to Start a Flink Cluster

There are three main startup modes:

Standalone : The most basic mode with full functionality, deployable on physical machines.

Cluster session : Deployed inside resource schedulers such as YARN or Kubernetes; jobs are submitted via JobManager. This mode has known bugs (e.g., FLINK‑11205) related to Metaspace leakage.

Per‑Job : Integrated in YARN; the cluster lifecycle is bound to the job, and resources are released when the job finishes.

JobManager Startup

Since Flink 1.5, JobManager includes a ResourceManager (managing Flink‑specific TaskManager resources) and a Dispatcher (providing RPC interfaces for clients). Each Flink job also has a JobMaster that manages its lifecycle.

TaskManager Startup

TaskManager provides basic RPC services for JobManager scheduling and later runs the subtasks assigned by JobManager.

Flink Job Startup

A Flink job consists of an execution environment (ENV) where users register their code logic. The ENV defines rules for building the DAG, such as source types, keyBy, etc., and can run in different environments.

Client Side

The client generates several layers:

User Rule : First abstraction from user code to DAG.

Transformation Layer : Maps user operators to transformations (e.g., SourceTransformation, OneInputTransformation). Physical transformations represent operators that require compute resources.

StreamGraph : Built from transformations, consisting of StreamNodes (vertices) and StreamEdges (edges).

JobGraph : The final object sent to JobManager, encapsulating the DAG and providing optimizations such as chaining to reduce serialization and network overhead.

JobManager Side

JobManager receives the JobGraph and creates an ExecutionGraph, which is the parallel version of the JobGraph. It contains ExecutionJobVertex (parallel vertices) and ExecutionVertex (the smallest execution unit, i.e., SubTask). TaskDeploymentDescriptors are generated for each vertex and sent to TaskManagers.

TaskManager Side

TaskManagers execute the received TaskDeploymentDescriptors, handling input and output gates and running the tasks. Communication between SubTasks on the same TaskManager uses InputGate and ResultPartition, while cross‑TaskManager communication relies on Netty.

ResultPartition notifies InputGate when data is available; if the downstream is busy, back‑pressure propagates upstream, which is Flink's native way of handling congestion.

The article concludes that while the core job flow is covered, many other aspects such as checkpointing and state storage are not discussed.

References

Flink official site: https://ci.apache.org/projects/flink/flink-docs-release-1.9/

FLIP proposals: https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

Jark's Blog: http://wuchong.me/

Yu Zhao's blog: http://chenyuzhao.me/

Flink official blog on network stack: https://flink.apache.org/2019/06/05/flink-network-stack.html