Upgrading Real-Time Computing Engine from Flink 1.10 to 1.13: Practices and Challenges

This article details the process of upgrading the real-time computing engine from Flink 1.10 to Flink 1.13 at Youzan. With the increasing demand for SQL capabilities and the need for containerization, the upgrade was necessary to leverage community support for on K8S capabilities and enhance the engine's functionality.

The article begins by outlining the background and motivation for the upgrade, highlighting the growing demand for SQL capabilities and the need for containerization. It then discusses the benefits of upgrading to Flink 1.13, including improvements in SQL syntax, time zone and time function optimizations, support for Window TVF syntax standardization, and enhanced Flink on Hive capabilities.

The article also covers the benefits of Flink on K8S, such as improved stability and maturity, real-time offline elastic scaling, and enhanced state retention and recovery features. It further discusses the support for upsert Kafka and richer format formats, as well as other benefits like improved memory metrics, backpressure visualization, CPU flame graph viewing, and historical exception support in the Web UI.

The article then delves into the practical aspects of the upgrade process, including the upgrade of custom connectors, UDF compatibility, SQL syntax conversion, and smooth task migration. It provides detailed examples and code snippets to illustrate the challenges and solutions encountered during the upgrade.

Finally, the article concludes by summarizing the successful upgrade of the real-time computing engine to Flink 1.13 and the migration of all Flink SQL tasks to the K8S cluster. It emphasizes the importance of SQL simplification and feature enhancement for future business scenarios and the readiness for elastic scaling.