Comprehensive Flink Interview Guide: Architecture, APIs, Operators, and Advanced Topics

Flink is a stream‑processing engine that offers distributed computation, fault tolerance, and a rich set of APIs: DataSet for batch, DataStream for streaming, and Table API/SQL for relational queries, supporting Java, Scala, and Python.

Compared with Spark Streaming, Flink uses an event‑driven model (no micro‑batches), has a different architecture (JobManager, TaskManager, Client), supports three time semantics (event, ingestion, processing), and provides watermarks to handle out‑of‑order events.

The component stack consists of Deployment (local, standalone/YARN, cloud), Runtime (core execution, scheduling), API (DataSet/DataStream), and Libraries (Flink ML, Gelly, CEP, etc.).

Key programming concepts include streams and transformations, sources and sinks, and a variety of operators such as map, flatMap, filter, keyBy, reduce, fold, window, windowAll, join, coGroup, connect, split, select, and iterate.

Partitioning strategies include Global, Shuffle, Rebalance, Rescale, Broadcast, Forward, Hash (KeyGroupStream), and Custom partitioners.

Parallelism can be set at operator, execution‑environment, client, or system level, with defaults configurable via flink-conf.yaml and ExecutionEnvironment.setParallelism().

Restart strategies supported are Fixed‑Delay, Failure‑Rate, No‑Restart, and Fallback, each configurable in the Flink configuration.

Flink provides a distributed cache similar to Hadoop's, and broadcast variables to share read‑only data across parallel tasks.

State backends (MemoryStateBackend, FsStateBackend, RocksDBStateBackend) store operator state for fault‑tolerant checkpoints.

Time handling includes event time, ingestion time, and processing time, with watermarks used to progress event‑time windows.

TableEnvironment manages catalogs, registers functions, and bridges DataStream/DataSet to tables for SQL execution.

SQL parsing uses Apache Calcite for validation and logical planning, followed by Flink‑specific optimization and code generation via Janino.

Flink unifies batch and stream processing by treating bounded streams as a special case of unbounded streams.

Data transmission relies on buffered network transfers managed by TaskManager, achieving high throughput.

Fault tolerance is achieved through lightweight asynchronous snapshots (Chandy‑Lamport algorithm) and a two‑phase commit protocol, guaranteeing exactly‑once semantics.

Memory management uses pre‑allocated MemorySegments for object storage, network buffers, and a large off‑heap memory pool.

Serialization is handled by type‑specific serializers (BasicTypeInfo, TupleTypeInfo, PojoTypeInfo, etc.) with Kryo as a fallback.

RPC communication between JobManager, TaskManager, and Dispatcher is built on Akka.

Back‑pressure is detected and mitigated by adjusting data flow rates, differing from Spark’s approach.

Operator chaining merges compatible operators into a single task to reduce thread switches, serialization, and latency.