Apache Flink at Kuaishou: Past, Present, and Future

Why Flink? Flink was chosen for sub‑second latency, rich windowing and state semantics, and unified batch‑stream capabilities, which simplify development and meet real‑time processing demands.

Flink Development at Kuaishou began in 2017 with pilot projects for live‑stream quality monitoring. By 2018, a surrounding ecosystem (metrics, monitoring, job‑hosting) was built, supporting CDN traffic scheduling, log splitting, and crash analysis. In 2019, stability and performance enhancements such as avalanche protection, flow control, hot‑parameter updates, the Slimbase state engine, and multi‑dimensional modeling were added, enabling large‑scale events and real‑time joins. Since 2020, the focus shifted to efficiency, adopting Flink SQL for batch‑stream integration, AI data flows, and real‑time reporting.

Business Data Flows involve three data sources (databases, server logs, client logs) ingested via Kafka, then processed in real‑time pipelines that feed downstream stores like Druid, ClickHouse, MySQL, Redis, and HBase. Specific scenarios include:

Real‑time ETL using Flink SQL, with schema managed by a metadata service.

Data integration across Kafka, HBase, Elasticsearch, Hive, and Redis via Flink’s rich source/sink connectors.

Real‑time reporting for large events, using a four‑layer pipeline (ODS → Sampling → Metric Calculation → Data Service).

Live‑stream quality monitoring and CDN traffic scheduling, producing DWD/DWS layers that feed OLAP stores and Redis for decision‑making.

AI feature processing: streaming feature computation, storage in a feature store and Hive, and index generation for recommendation systems.

Technical Innovations focus on the self‑developed Slimbase state engine, which consists of an interface layer, a middle KV‑cache layer (high‑speed KV + Chunk for space efficiency), and a distributed file‑system layer with compaction and block‑level caching. Benchmarks against RocksDB show 3‑9× read/write speedups on 0.5 M records, 2‑6× on 15 M, and comparable performance on 50 M with room for further optimization.

Stability Improvements address hardware failures, dependent service outages, and overload scenarios. Faster failure detection (10 s) is achieved with the Hawk Service, and resource redundancy is reserved to cut recovery time from ~90 s to ~15 s. Kafka failover strategies and source coordination mechanisms (SourceCoordinator, HealthyCoordinator) regulate consumption rates to prevent back‑pressure and hardware saturation.

Balancing and Resource Management tackle load imbalance across Yarn nodes and TaskManagers through improved scheduling, consumption‑aware resource requests, and dynamic scaling. Strategies include task‑level balancing, real‑time resource usage feedback to Yarn, and asynchronous job adjustments during cluster scaling.

Future Plans aim to expand batch‑stream integration, deepen AI data‑flow usage, enhance stability (fast failover), and implement elastic scaling for Flink jobs.