Remote StateBackend for Flink: Design, Optimizations, and Cloud‑Native Migration

To enable Bilibili’s cloud‑native migration, the team built a RemoteStateBackend that moves Flink’s keyed state to the Taishan KV store, using deterministic KeyGroup placement, per‑shard snapshots, asynchronous write batching, off‑heap caching with Bloom‑filter filtering, and a fixed‑size memory model, which together reduce checkpoint overhead, improve disk utilization, and accelerate rescaling for more than one hundred production jobs.

CloudNativeFlinkPerformanceOptimization

0 likes · 18 min read

Remote StateBackend for Flink: Design, Optimizations, and Cloud‑Native Migration

Big Data Technology & Architecture

Mar 27, 2023 · Big Data

Key Updates in Apache Flink 1.17: Batch and Streaming Enhancements

The article reviews Apache Flink 1.17's major batch and streaming improvements, including new Delete/Update APIs, performance boosts, SQL client gateway, checkpoint and watermark enhancements, StateBackend upgrades, and practical use‑case scenarios for data engineers.

Apache FlinkBatch ProcessingBig Data

0 likes · 7 min read

Key Updates in Apache Flink 1.17: Batch and Streaming Enhancements

Big Data Technology & Architecture

Nov 16, 2021 · Big Data

Flink Checkpoint, Backpressure, and Memory Tuning Guide

This article provides a comprehensive guide on optimizing Flink checkpoints, diagnosing and alleviating backpressure, and fine‑tuning memory configurations—including process, heap, off‑heap, managed, and network memory—to improve job stability and performance in large‑scale streaming applications.

CheckpointFlinkMemory Tuning

0 likes · 25 min read

Flink Checkpoint, Backpressure, and Memory Tuning Guide