Understanding Flink Execution Resources: Operator Chains, Task Slots, Slot Sharing and CoLocation

Flink treats the resources used to execute tasks as logical concepts and provides several core abstractions such as Slot, SlotSharingGroup, CoLocationGroup, and OperatorChain to manage and isolate these resources.

Operator Chains – For efficient distributed execution, Flink links subtasks of operators into a single task that runs in one thread. Chaining reduces thread switches, serialization, buffer exchanges, and latency while increasing throughput. The article lists the strict conditions required for two operators to be chained (matching parallelism, downstream indegree = 1, same slot group, chain strategy ALWAYS/HEAD, forward partitioning, and no user‑disabled chaining). The API methods startNewChain() , disableChaining() and StreamExecutionEnvironment.disableOperatorChaining() control chaining behavior.

The internal implementation uses the OperatorChain class, which hides the internal ChainOperator and HeadOperator details. Data flowing inside a chain is passed directly via ChainingOutput without serialization or network transfer, as shown in the provided source code snippet.

private static class ChainingOutput<T> implements Output<StreamRecord<T>> { // registered downstream operator protected final OneInputStreamOperator<T, ?> operator; public ChainingOutput(OneInputStreamOperator<T, ?> operator) { this.operator = operator; } @Override public void collect(StreamRecord<T> record) { try { operator.setKeyContextElement1(record); operator.processElement(record); } catch (Exception e) { throw new ExceptionInChainedOperatorException(e); } } ... }

Task Slot – A TaskSlot represents a fixed‑size subset of a TaskManager’s resources. Each slot isolates memory for tasks but does not provide CPU isolation. By adjusting the number of slots, users control task isolation and resource sharing. Examples illustrate how a 5‑parallel WordCount job can be mapped onto two TaskManagers with three slots each.

SlotSharingGroup and CoLocationGroup – By default, operators belong to the default sharing group, allowing their subtasks to share a slot. The SlotSharingGroup class enables slot sharing, while CoLocationGroup forces subtasks to be placed on the same slot (used mainly for iterative streams). The article shows how to assign a custom sharing group via someStream.filter(...).slotSharingGroup("group1") .

The allocation algorithm is demonstrated with a series of diagrams: source, flatMap, and key‑aggregation/sink subtasks are assigned to SimpleSlot or SharedSlot instances according to the constraints that subtasks of the same operator cannot reside in the same SharedSlot and that scheduling follows topological order.

In summary, Flink’s execution resources revolve around TaskSlots, operator chaining, and slot sharing groups, which together enable high‑throughput, low‑latency stream processing while maximizing resource utilization.