Flink on Kubernetes: Architecture, Deployment Modes, and Operational Guide

Flink on Kubernetes Overview

Flink is a leading stream processing engine that also supports batch and machine learning workloads. Combining it with Kubernetes (K8s) provides powerful container orchestration and resource management, surpassing YARN and Mesos.

Flink Cluster Architecture

In a Flink cluster there is one JobManager and multiple TaskManagers. The client submits a job to the JobManager, which schedules tasks to TaskManagers. TaskManagers exchange data via streams and report heartbeats and metrics back to the JobManager.

Running Flink on K8s

Two deployment modes are available: Standalone and native (K8s native). Both support Session and Per‑Job execution models.

Standalone Mode

Deploy a Flink cluster by creating a ConfigMap for configuration files, a JobManager Deployment and Service, and a TaskManager Deployment. Example commands:

kubectl create -f flink-configuration-configmap.yaml

kubectl create -f jobmanager-deployment.yaml

kubectl create -f jobmanager-service.yaml

kubectl create -f taskmanager-deployment.yaml

Submit jobs via port‑forwarding to the JobManager Service:

kubectl port-forward service/flink-jobmanager 8081:8081

./bin/flink run -d -m localhost:8081 ./examples/streaming/TopSpeedWindowing.jar

Delete the cluster with:

kubectl delete -f jobmanager-deployment.yaml
kubectl delete -f taskmanager-deployment.yaml
kubectl delete -f jobmanager-service.yaml
kubectl delete -f flink-configuration-configmap.yaml

Native Mode

From Flink 1.9 onward, Flink includes a native K8s client that can request resources directly from the cluster. Preparation includes a valid kubeconfig, DNS enabled, and appropriate permissions.

Native mode also supports Session and Per‑Job models. Session mode can be started with:

./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dtaskmanager.memory.process.size=4096m -Dkubernetes.taskmanager.cpu=2 -Dtaskmanager.numberOfTaskSlots=4 -Dresourcemanager.taskmanager-timeout=3600000

Then submit a job:

./bin/flink run -d -e kubernetes-session -Dkubernetes.cluster-id=<ClusterId> examples/streaming/WindowJoin.jar

Per‑Job mode (experimental, supported from Flink 1.11) builds a custom Docker image containing the user JAR and runs a dedicated Flink job cluster:

sh build.sh --from-release --flink-version 1.7.0 --hadoop-version 2.8 --scala-version 2.11 --job-jar ~/flink/flink-1.7.1/examples/streaming/TopSpeedWindowing.jar --image-name topspeed

docker tag topspeed zkb555/topspeedwindowing
docker push zkb555/topspeedwindowing

# Deploy service
kubectl create -f job-cluster-service.yaml
# Deploy JobManager
FLINK_IMAGE_NAME=zkb555/topspeedwindowing:latest FLINK_JOB=org.apache.flink.streaming.examples.windowing.TopSpeedWindowing FLINK_JOB_PARALLELISM=3 envsubst < job-cluster-job.yaml.template | kubectl create -f -
# Deploy TaskManager
FLINK_IMAGE_NAME=zkb555/topspeedwindowing:latest FLINK_JOB_PARALLELISM=4 envsubst < task-manager-deployment.yaml.template | kubectl create -f -

Both modes allow dynamic scaling: the native resource manager can launch additional TaskManager pods when more slots are needed.

Conclusion

Native mode offers greater flexibility by not requiring a fixed number of TaskManagers, making it preferable to Standalone for production workloads. Choose Session mode for short‑lived, frequently submitted jobs, and Per‑Job mode for long‑running streaming jobs.