Understanding Flink TaskManager Memory Model (Post‑1.10)

First, here is the official Flink (post‑1.10) memory model diagram:

Next, the parameter information of a running Flink job and its TaskManager memory:

Now let’s interpret the diagrams:

From the startup configuration we can see that the maximum and initial heap memory are both 1.5 GB, DirectMemory is about 471 MB, and the Flink task runs inside a single TaskManager JVM process. The following sections analyze the internal memory structure of this process.

Total Process Memory

The TaskManager memory consists of five parts: heap memory, off‑heap memory, direct memory, Metaspace memory, and JVM overhead memory. Their sum is the Total Process Memory, which can be set via taskmanager.memory.process.size in flink-conf.yml. In the example the total memory is 4 GB.

Total Flink Memory

Total Flink Memory includes all memory used by Flink (excluding JVM Metaspace and overhead). It is divided into four blocks: Flink framework memory (heap/off‑heap), managed memory (off‑heap), network cache (off‑heap), and task memory (heap/off‑heap). It can be configured with taskmanager.memory.flink.size. The official recommendation is not to configure both Process Memory and Flink Memory simultaneously.

# It is not recommended to set both 'taskmanager.memory.process.size' and Flink memory.

JVM Heap

The JVM heap is split into two major parts: the framework heap required by Flink itself and the task heap used by user tasks.

Framework Heap

Framework Heap is reserved for Flink’s internal use and is not used for executing tasks. Its size is set via taskmanager.memory.framework.heap.size.

Task Heap

Task Heap Memory is the heap space dedicated to executing Flink tasks (user code, custom data structures). Its size is configured with taskmanager.memory.task.heap.size.

Off‑Heap Memory

Managed Memory

Managed Memory is off‑heap memory directly managed by Flink for sorting, hash tables, intermediate result caching, and RocksDB state backend. It is set with taskmanager.memory.managed.size. By default it is not configured; otherwise it is derived from taskmanager.memory.managed.fraction (default 0.4) multiplied by Total Flink Memory.

Direct Memory

Framework Off‑heap Memory

Flink framework off‑heap memory defaults to 128 MB and can be set via taskmanager.memory.framework.off-heap.size. Modification is not recommended.

Task Off‑heap Memory

Task Off‑heap Memory is used when native methods are called from Flink applications. It is configured with taskmanager.memory.task.off-heap.size and defaults to 0.

Network Memory

Network Memory is required for shuffle, broadcast, and data transfer between tasks and external systems. Its size is determined by three parameters:

taskmanager.memory.network.min – minimum network memory

taskmanager.memory.network.max – maximum network memory

taskmanager.memory.network.fraction – proportion of Total Flink Memory (default 0.1). If the calculated value exceeds the min‑max range, the min‑max values take precedence. If min and max are equal, the network memory is fixed.

JVM Metaspace Memory

Since JDK 8, the permanent generation has been removed and class metadata resides in Metaspace. Its size is set via taskmanager.memory.jvm-metaspace.size, defaulting to 256 MB.

JVM Overhead Memory

This reserves native memory for thread stacks, code caches, etc. It is configured with three parameters:

taskmanager.memory.jvm-overhead.min – minimum overhead (default 192 MB)

taskmanager.memory.jvm-overhead.max – maximum overhead (default 1 GB)

taskmanager.memory.jvm-overhead.fraction – proportion of Total Process Memory (default 0.1)

Summary

Combining the official TaskManager memory model with the real‑world example, the allocation diagram of each memory component is shown below: