Comprehensive Flink Interview Guide: Core Concepts, Advanced Topics, and Source‑Code Insights

2019 marked a turning point for real‑time big‑data computing when Alibaba open‑sourced Blink, a Flink fork, sparking a rivalry between Spark and Flink. Flink’s event‑driven streaming model and high performance have made it a hot topic in big‑data interviews.

Part 1 – Core Concepts and Basics

Flink is a distributed processing engine for both bounded and unbounded streams, offering stateful computation, fault tolerance, and resource management. It provides high‑level APIs: DataSet (batch), DataStream (stream), and Table (SQL‑like). Domain libraries include Flink‑ML and Gelly (graph processing). Key features are high throughput, low latency, exactly‑once semantics, flexible windowing, back‑pressure handling, lightweight snapshots, and support for both batch and streaming.

Compared with Spark Streaming, Flink uses a true streaming architecture (event‑driven) while Spark relies on micro‑batches. Their runtime models differ: Flink generates a StreamGraph → JobGraph → ExecutionGraph, whereas Spark builds a DStreamGraph and schedules jobs via a DAG.

Flink’s component stack consists of Deploy (deployment modes), Runtime (core execution), API (DataSet/DataStream/Table), and Libraries (ML, Gelly, CEP, etc.). Flink can run independently of Hadoop but integrates with YARN, HDFS, HBase, etc.

Part 2 – Advanced Topics

Flink supports batch‑stream convergence, efficient data exchange via buffered batches, and robust fault tolerance through checkpointing and two‑phase commit. It offers multiple restart strategies (fixed‑delay, failure‑rate, none, fallback) and a distributed snapshot algorithm based on Chandy‑Lamport.

State backends include MemoryStateBackend, FsStateBackend, and RocksDBStateBackend. Time semantics cover processing time, event time, and ingestion time, with watermarks handling out‑of‑order events. Windowing supports time‑ and count‑based tumbling and sliding windows.

Partitioning strategies (Global, Shuffle, Rebalance, Rescale, Broadcast, Forward, KeyGroup, Custom) are illustrated, with a custom partitioner example:

static class CustomPartitioner implements Partitioner<String> {
    @Override
    public int partition(String key, int numPartitions) {
        switch (key) {
            case "1": return 1;
            case "2": return 2;
            case "3": return 3;
            default:  return 4;
        }
    }
}

Parallelism can be set at operator, execution environment, client, or system level (priority: operator > env > client > system). Slots represent the concurrency capacity of a TaskManager, while parallelism determines how many slots are actually used.

Flink’s memory management uses Network Buffers, a MemoryManager pool, and user‑code memory, often leveraging off‑heap buffers. Serialization relies on Flink’s own TypeInformation hierarchy (BasicTypeInfo, TupleTypeInfo, PojoTypeInfo, etc.) rather than Java’s default serialization.

Operator chaining merges compatible operators into a single task to reduce thread switches, serialization, and network overhead. Conditions for chaining include matching parallelism, single downstream input, same slot group, ALWAYS chain strategy, forward partitioning, and no user‑disabled chaining.

Flink 1.9 introduced Hive support, UDFs, SQL optimizations (TopN, GroupBy), improved checkpoint/savepoint handling, state query APIs, and other enhancements.

Part 3 – Source‑Code Deep Dive

The job submission pipeline transforms user code into a StreamGraph, then a JobGraph (optimized and resource‑aware), and finally an ExecutionGraph (runtime‑ready). JobManager acts as the master, handling job registration, scheduling, checkpoint coordination, and communication with TaskManagers via Akka.

TaskManagers are workers that register with the JobManager, acquire TaskSlots, and execute tasks. Slots isolate resources; multiple slots can share a JVM, while a single slot runs a single task.

Data flow uses MemorySegment (32 KB blocks), Buffer, and StreamRecord abstractions to move objects efficiently between operators and the network.

Distributed snapshots insert barriers at sources; when all downstream operators receive a barrier, a consistent snapshot is taken, enabling exactly‑once recovery.

Flink SQL parsing and planning rely on Apache Calcite: SQL is parsed into a logical plan, optimized, and then translated into a physical plan that is code‑generated (via Janino) and executed on the DataStream runtime.

Example of registering a cached file (Scala):

val env = ExecutionEnvironment.getExecutionEnvironment

// register a file from HDFS
env.registerCachedFile("hdfs:///path/to/your/file", "hdfsFile")

// register a local executable file
env.registerCachedFile("file:///path/to/exec/file", "localExecFile", true)

// define your program and execute
val input: DataSet[String] = ...
val result: DataSet[Integer] = input.map(new MyMapper())

env.execute()

The article concludes with a reminder to like, share, and collect the post.