Flink 1.16 Highlights: Adaptive Batch Scheduling, Speculative Execution, Hybrid Shuffle, Dynamic Partition Pruning, Hive SQL Migration, Checkpoint Enhancements, CDC Integration, and Table Store

Unified API, Compute, and Storage

Flink 1.16 provides a unified programming model that allows a single application to run in both streaming and batch environments, supports multiple data sources, and offers a common storage layer (Table Store) compatible with both Flink and Spark.

Adaptive Batch Scheduler

The Adaptive Batch Scheduler automatically determines the optimal parallelism for batch jobs, simplifying resource allocation and improving performance.

Speculative Execution

Speculative Execution detects and mitigates hotspot machines that degrade job performance; it is a new mechanism in Flink 1.16, currently supported at the source level.

Hybrid Shuffle

Hybrid Shuffle adapts to resource availability: when resources are abundant it uses a fully streaming shuffle, and when resources are constrained it falls back to a stable batch‑style shuffle, providing seamless performance without user intervention.

Dynamic Partition Pruning

Dynamic Partition Pruning filters out irrelevant partitions early in the execution plan, reducing data scanned and accelerating query processing.

Hive SQL Migration

Motivated by the need to attract offline data‑warehouse users and lower the barrier for Flink development, Flink 1.16 improves Hive SQL compatibility from 85% to 94.1% (based on the Hive qtest 12k suite) and reuses Hive syntax through a dedicated parser and relational node translation.

Checkpoint Evolution

Checkpointing has progressed from lightweight asynchronous snapshots in 0.9, through RocksDB StateBackend support (1.0), incremental checkpoints (1.3), unaligned checkpoints (1.11, 1.13), buffer debloating (1.14), to the changelog backend introduced in 1.15/1.16, which decouples state backend uploads from the changelog and further reduces checkpoint overhead.

Flink CDC + Kafka Integration

Flink CDC captures change data from databases and, combined with Kafka, enables real‑time data pipelines and analytics; the article outlines typical use cases and a demo architecture.

Flink Table Store

Table Store serves as a versatile storage solution for both real‑time and offline workloads, with examples of typical application scenarios and a demonstration of its capabilities.