Understanding Full GC, Data Skew, and Parallelism in Flink Tasks

The author answers several practical and interview‑related questions about Flink, noting that a QR code at the end links to a knowledge community.

About Full GC : Monitoring Full GC in Flink TaskManagers is critical because frequent GC can slow processing, cause TaskManager loss, fail‑over, or even OOM. Prometheus is commonly used to track Full GC count, but a lower count is not always better; both the number of collections and their duration should stay within reasonable ranges.

For detailed GC diagnostics, you can log into the TaskManager host and use tools such as Arthas to view thread stacks, flame graphs, and other runtime information.

Task data distribution : Flink displays input and output metrics per TaskManager, making it easy to spot data skew by comparing data volumes across managers. Slight skew is acceptable if it does not cause back‑pressure, but severe skew may require adding resources globally, which can lower overall utilization.

Handling data skew and parallelism : Increasing parallelism alone does not solve skew because the root cause is improper data distribution. Fixing the distribution (e.g., setting StreamPartitioner to rebalance) can alleviate skew. Adjusting parameters such as Redistributing also helps.

Kafka partition and consumer parallelism : It is strongly recommended to set the consumer parallelism equal to the number of Kafka partitions and to use a rebalance partitioner so that data is evenly spread across Flink tasks.

Redistributing parameter : If there are no special requirements, set this parameter to rebalance to avoid uneven data distribution.

The article concludes with a promotional block containing a QR code and numerous links to big‑data learning resources, interview guides, and related articles.