0 views collected around this technical thread.
In large‑scale Flink double‑stream joins, terabyte‑sized RocksDB state caused severe compaction latency and CPU spikes, but enabling RocksDB BlobDB KV‑separation (and an inner‑compaction patch) dramatically shrank SST files, reduced read/write latencies to sub‑millisecond levels, and cut CPU spikes by about half.
This article explains how to optimize Flink’s RocksDB state backend for large‑scale streaming jobs, covering state types, enabling latency tracking, incremental checkpoints, predefined options, and advanced memory and thread settings, with practical configuration examples and performance comparisons.
Bilibili’s FlinkSQL team has enhanced the Flink engine—now based on 1.11 with back‑ported 1.15 features—by adding Delay‑Join, table‑valued functions, projection‑push‑down, UDF and object reuse, automatic mini‑batch/two‑phase aggregation, key‑group skew fixes, connector slot‑groups, real‑time projection with Hudi, and RocksDB state‑performance tweaks, while planning remote state backends and deeper stream‑batch integration.
Flink 1.13 introduces passive scaling that lets users adjust parallelism to resize jobs, adds visual tools such as load/back‑pressure charts, CPU flame graphs, and state‑backend metrics for deeper performance insight, and includes numerous community optimizations for easier upgrades and operation.
This article describes how Kuaishou leverages Apache Flink for large‑scale real‑time multi‑dimensional analytics, details the architecture of its analytics platform using Kudu storage and KwaiBI, and introduces SlimBase—a lightweight, embedded shared state backend that replaces RocksDB to reduce I/O, latency, and CPU overhead.
This article presents Kuaishou's extensive use of Apache Flink for real-time multi-dimensional analytics, detailing the platform's architecture, cluster scale, data processing pipelines, the design of a shared state storage engine called SlimBase, and performance improvements achieved through replacing RocksDB with a customized HBase‑based solution.
This article describes the motivation, technology selection, implementation details, and practical challenges of building a real‑time data warehouse using Flink, covering stream ingestion, data cleaning, dimension‑table joins, state backend choices, and operational lessons for large‑scale streaming pipelines.
This article describes the motivation, technology selection, implementation details, and encountered challenges of building a real‑time data warehouse using Flink, covering streaming computation, code examples, dimension‑table caching, state backend choices, and best practices for production deployment.