Optimizing Flink Real-Time Task Resources: Memory and Message Processing Perspectives

Background

With Flink moving to Kubernetes (k8s) and the completion of real‑time cluster migration, many Flink jobs now run on k8s, improving elasticity during peak traffic and reducing operational costs. However, users often over‑allocate resources, e.g., configuring 16 parallelism when 4 is sufficient, leading to wasted compute resources.

1. Flink Computing Resource Types and Optimization Ideas

1.1 Resource Types

Flink tasks consume five main resource categories: memory, local (or cloud) disk storage, external storage (HDFS, S3, HBase, MySQL, Redis), CPU, and network cards. In practice, memory and CPU are the primary bottlenecks, so the optimization focuses on these two.

1.2 Optimization Thinking

The approach consists of two angles: analyzing heap memory via GC logs and evaluating message‑processing capacity to ensure CPU is used efficiently. By combining memory metrics and concurrency analysis, a recommended resource preset is derived and communicated with business owners for adjustment.

1.2.1 Memory‑Centric Analysis

GC logs provide per‑GC heap region changes and the remaining old‑generation space after a Full GC. Using the open‑source GC Viewer , we extract metrics such as total heap size, young/old generation usage, and Full GC pause times. According to the rule from "Java Performance Optimization" the recommended heap size is 3‑4× the remaining old‑gen space after Full GC.

When the recommended heap differs significantly from the actual allocation, the task’s memory configuration can be reduced to save resources.

1.2.2 Message‑Processing Capacity

We compare the Kafka topic input rate (records per second) with the processing capacity of the slowest operator/task. The slowest task’s concurrency (P), its per‑record processing time (T), and its output rate (O) are used to decide whether to increase, decrease, or keep the parallelism unchanged.

Three cases are considered:

If O ≈ S and (1/T)·P » S, reduce parallelism.

If O ≈ S and (1/T)·P ≈ S, keep parallelism.

If O « S and (1/T)·P « S, increase parallelism.

A periodic detection job flags tasks that repeatedly fall into case 1, triggering an alert for manual optimization.

2. Memory‑Centric Practice

2.1 GC Collector Choice

Flink runs on Java 1.8; with CPU limits of 0.6‑1 core per pod, the Serial Old collector is chosen for the old generation to avoid multi‑thread overhead.

2.2 Obtaining GC Logs

GC logs are collected from the top‑ranked TaskManagers (by young‑GC count) via Flink REST API, then stored in Kafka through a Filebeat‑based pipeline, and finally downloaded via an internal log service.

2.3 Analyzing Logs with GC Viewer

After downloading a TaskManager’s gc.log, run: java -jar gcviewer-1.37-SNAPSHOT.jar gc.log summary.csv The resulting CSV contains fields such as RunHours, YGSize, OGSize, YGCoun, FGCount, FGAllTime, Throughput, RecHeap, RecNewHeap, and RecOldHeap, which are used to compute the recommended heap sizes.

3. Message‑Processing Perspective

3.1 Obtaining Kafka Topic Input Rate

For Flink SQL jobs, the topic is extracted from the CREATE TABLE statement. For Flink JAR jobs, a lineage service builds a PackagedProgram, extracts the StreamGraph, and reflects the source function to retrieve the Kafka topic.

3.2 Automated Detection of the Slowest Task

We added a custom metric taskOneRecordDealTime to each Task via Flink’s metric system. Using the Flink REST API, we enumerate all vertices, fetch the metric for each parallel task, and record the maximum per‑vertex value. The vertex with the highest value is identified as the bottleneck.

API examples:

base_flink_web_ui_url/jobs/:jobid

.../vertices/:vertexid/metrics?get=taskOneRecordDealTime

4. Practical Resource Optimization at Youzan

The platform runs a daily scan of active Flink jobs. If the recommended heap size deviates by a large factor from the actual allocation, an alert is sent to administrators. After confirming that the message‑processing capacity is also reasonable, the admin may reduce parallelism after business discussion.

5. Conclusion

Youzan’s real‑time computing platform has taken the first steps toward automated Flink resource optimization, combining memory‑based recommendations and message‑processing analysis. Future work aims to fully automate the entire optimization loop, leveraging historical usage patterns to dynamically adjust resources and improve cluster utilization.