How Kuaishou Boosted Flink SQL Performance with Window Extensions and State Optimizations

Flink SQL at Kuaishou

After more than a year of promotion, internal users' acceptance of Flink SQL grew; this year SQL jobs account for 60% of new Flink jobs, a two‑fold increase from last year, with peak throughput reaching 600 million rows per second.

Feature Extensions

Group Window Aggregate Extension

Multidimensional Analysis

To support internal business needs, Kuaishou added multidimensional analysis to Group Window Aggregate, supporting standard GROUPING SETS, ROLLUP, and CUBE clauses, as well as various window types such as tumbling, sliding, and session windows.

For example, to compute cumulative UV by topic and total dimensions, the SQL GROUP BY clause combines a CUMULATE window function with a GROUPING SETS clause.

High‑order Window Functions

Developers often need a curve where each point represents the cumulative metric from midnight to the current time. Two solutions were considered: an infinite‑stream aggregation (which produces an update stream) and a one‑day sliding window with early fire (which suffers from inconsistent values and “pits” in the curve).

To solve these issues, Kuaishou introduced a CUMULATE window that uses the window end time as the x‑axis and the cumulative value as the y‑axis, ensuring smooth curves even during failover or historical back‑tracking.

The CUMULATE window requires three mandatory parameters—time attribute column, window step, and max size—and an optional offset parameter.

With a step of one minute and a max size of three minutes, a timestamp at 0:30 belongs to three overlapping windows (0‑1, 0‑2, 0‑3), enabling accurate per‑minute cumulative UV calculations.

Window Table‑valued Function Extension

Rich Window TVF Operators

Building on the Flip‑145 Window TVF proposal, Kuaishou enriched the operator set with TopN, join, and deduplication capabilities, and added a standalone Window TVF query syntax, allowing more complex business logic in Flink SQL.

Window Offset

The optional window offset adjusts how windows are aligned without affecting watermarks. Positive values shift windows to the right, negative values to the left, and different offsets can produce equivalent window boundaries.

Batch Mode Support

Kuaishou also added batch‑mode support for Window TVF by introducing a windows operator that attaches window attributes to input rows, allowing downstream operators to reuse existing batch implementations (HashAgg, SortAgg, HashJoin, SortMergeJoin) for better throughput.

Performance Optimizations

Aggregation State Reuse

Kuaishou tackled distinct‑state duplication by manually rewriting SQL to pivot multiple count‑distinct aggregations into separate filter conditions, then using a custom table function to unpivot the results. This reduces state size but requires manual SQL changes and a custom function.

To simplify this, a new SQL expression allows developers to declare enumerated group‑key values; the optimizer automatically rewrites the query, performs column‑to‑row conversion, and reuses the distinct map state.

Mini‑Batch Optimization for Dimension Table Joins

A Mini‑Batch hint (time interval or row count) batches dimension‑table lookups, reducing RPC calls. The hint is generic and can also be applied to aggregation mini‑batching.

Stability Improvements

Data Skew Mitigation for Group Window Aggregate

To handle data skew in high‑traffic scenarios (e.g., popular live streams), Kuaishou added Mini‑Batch, Local‑Global, and Split‑Distinct optimizations to tumbling windows and similar window types, improving both latency and throughput.

Aggregate State Compatibility

When business requirements change (adding or removing metrics), Flink’s simple state‑compatibility check can block upgrades. Kuaishou introduced an aggregate‑state compatibility scheme that allows adding new aggregate functions at the end of the list and removing any existing ones without changing order, while preserving state TTL and handling early/late fire retract messages.

The scheme stores a mapping from metric identifiers to state types in state metadata, enabling forward compatibility across versions.

Future work aims at a “ultimate” state‑compatibility solution that lets users add, delete, or modify arbitrary state entries in a Savepoint, without requiring code changes, supporting all scenarios.

Future Outlook

Kuaishou will continue expanding Stream SQL capabilities, improving performance, and exploring more state‑compatibility scenarios. In the batch domain, it plans to enhance Flink Batch SQL with speculative execution and adaptive query optimization, while further promoting data‑lake and real‑time warehouse adoption across business lines.