Master Flink on Kubernetes: Step‑by‑Step Deployment Guide

Overview

Flink is a streaming data‑flow engine that can run both stream and batch jobs on a unified runtime.

Official resources: Flink website , documentation, GitHub repository.

Flink runtime modes on YARN

Session mode

Per‑Job mode (deprecated after Flink 1.15)

Application mode

Deploy Flink on Kubernetes – practical steps

1. Download Flink

wget https://dlcdn.apache.org/flink/flink-1.14.6/flink-1.14.6-bin-scala_2.12.tgz

2. Build a base Docker image

docker pull apache/flink:1.14.6-scala_2.12
docker tag apache/flink:1.14.6-scala_2.12 myharbor.com/bigdata/flink:1.14.6-scala_2.12
docker push myharbor.com/bigdata/flink:1.14.6-scala_2.12

3. Create a Flink session cluster

Deploy the JobManager, TaskManagers and expose the REST/UI ports.

Run JobManager container

Run TaskManager containers

Expose ports 6123 (RPC), 6124 (Blob), 8081 (Web UI)

Native Kubernetes mode – configuration

Key components to configure:

JobManager deployment

TaskManager deployment

ConfigMap with flink-conf.yaml and logging properties

Service definitions for JobManager, REST, and TaskManager query state

Example Dockerfile for a custom image

FROM myharbor.com/bigdata/centos:7.9.2009
USER root
RUN yum install -y vim tar wget curl rsync bzip2 iptables tcpdump less telnet net-tools lsof
RUN rm -f /etc/localtime && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
ADD jdk-8u212-linux-x64.tar.gz /opt/apache/
ADD flink-1.14.6-bin-scala_2.12.tgz /opt/apache/
ENV FLINK_HOME /opt/apache/flink-1.14.6
ENV JAVA_HOME /opt/apache/jdk1.8.0_212
ENV PATH $JAVA_HOME/bin:$PATH
RUN mkdir -p $FLINK_HOME/usrlib/
COPY docker-entrypoint.sh $FLINK_HOME/
ENTRYPOINT ["/opt/apache/docker-entrypoint.sh"]
CMD ["help"]

Deploy the image

docker build -t myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12 . --no-cache
docker push myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12

Kubernetes resources

Create namespace and service account:

kubectl create ns flink
kubectl create serviceaccount flink-service-account -n flink
kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit --serviceaccount=flink:flink-service-account

ConfigMap ( flink-configuration-configmap.yaml) contains the core Flink configuration and log settings.

Service definitions ( jobmanager-service.yaml, jobmanager-rest-service.yaml, taskmanager-query-state-service.yaml) expose the necessary ports, using NodePort for external access.

Deployments ( jobmanager-session-deployment-non-ha.yaml, taskmanager-session-deployment.yaml) reference the ConfigMap and run the containers as non‑root user 9999.

Submit a job

./bin/flink run -m local-168-182-110:30081 ./examples/streaming/WordCount.jar

Monitor the cluster with kubectl get pods -n flink -o wide and access the Web UI at http://<em>node-ip</em>:30081/#/overview.

Cleanup

kubectl delete -f jobmanager-service.yaml -n flink
kubectl delete -f flink-configuration-configmap.yaml -n flink
kubectl delete -f taskmanager-session-deployment.yaml -n flink
kubectl delete -f jobmanager-session-deployment.yaml -n flink
kubectl delete ns flink --force

Application mode (recommended)

Build an application‑specific image that bundles the user JAR, then launch with flink run-application targeting Kubernetes.

./bin/flink run-application \
  --target kubernetes-application \
  -Dkubernetes.cluster-id=my-first-application-cluster \
  -Dkubernetes.container.image=myharbor.com/bigdata/flink-application:1.14.6-scala_2.12 \
  -Dkubernetes.jobmanager.replicas=1 \
  -Dkubernetes.namespace=flink \
  -Dkubernetes.jobmanager.service-account=flink-service-account \
  -Dexternal-resource.limits.kubernetes.cpu=2000m \
  -Dexternal-resource.limits.kubernetes.memory=2Gi \
  -Dexternal-resource.requests.kubernetes.cpu=1000m \
  -Dexternal-resource.requests.kubernetes.memory=1Gi \
  -Dkubernetes.rest-service.exposed.type=NodePort \
  local:///opt/flink/usrlib/TopSpeedWindowing.jar

Access the UI via the NodePort defined in jobmanager-rest-service.yaml.

Delete the application cluster

kubectl delete deployment/my-first-application-cluster -n flink
kubectl delete ns flink --force

References

Official Flink documentation, Kubernetes deployment guides, and community examples.