Optimizing Flink Sliding Windows for Super Long Time Ranges

This article discusses performance issues with Flink sliding windows in super long time ranges and presents solutions implemented at Youzan. The author explains why Flink's performance degrades significantly in these scenarios and how they addressed the problem.

The article begins by introducing Flink's window functionality, which supports various window types including sliding windows, tumbling windows, and session windows. The complexity of supporting these different window types can lead to performance issues, particularly with super long sliding windows.

The core of Flink's window implementation involves three key components: WindowAssigner, Trigger, and Evaluation Function. The WindowAssigner determines which windows an element belongs to, the Trigger decides when to fire the window, and the Evaluation Function defines how to compute results within the window.

For sliding windows, each element can belong to multiple windows. The number of windows an element belongs to can be calculated as n = window length / slide step, typically ranging from [n, n+1]. The article provides detailed code examples showing how SlidingEventTimeWindows assigns windows to elements based on their timestamps.

The performance issues arise from two main problems: state writing overhead and timer management. When using RocksDB as the state backend, writing and reading operations are slower than in-memory operations. For example, with a one-hour window length and five-minute step size, each element needs to be written to 288 different window states, resulting in throughput degradation of over 288 times compared to tumbling windows.

The timer management issue stems from the need to register timers for both triggering window output and cleaning up window state. Each key-window pair requires at least two timers, and while timers can be deduplicated, the read operations still impact throughput.

The solution involves time slicing to reduce the n-fold state read/write operations. By finding the greatest common divisor of window length and step size, elements can be stored in overlapping windows, ensuring each element only updates one overlapping window when entering the window operator. This approach significantly reduces state writing costs.

The article provides detailed mathematical analysis of read optimization under different data distribution scenarios: dense distribution, sparse distribution, and periodic batch appearance. The analysis shows that read optimization performance varies significantly depending on data characteristics and window configurations.

For the timer issue, the solution is simpler: when allowLateness is set to 0, only one timer needs to be registered for both triggering output and cleaning up state, potentially reducing timer write costs by up to 50%.

The article concludes by mentioning that while this optimization is effective, it's more of a workaround than an elegant solution. Youzan is also exploring other approaches including using Druid for pre-aggregation and building real-time metric services that store fine-grained results in online storage for read-time aggregation.

The optimization has shown 3-8x performance improvements in production, depending on the overlap coefficient. The author also mentions that Youzan's big data team is hiring and provides contact information for interested candidates.