Fine‑grained Configuration, State Migration, and Debugging Techniques for Flink SQL at Meituan

Meituan currently runs over 100 Flink SQL jobs serving more than 5,000 SQL tasks, accounting for 35% of all Flink workloads and growing at 115% year‑over‑year. The rapid increase brings challenges such as coarse‑grained StateTTL and concurrency configuration, inability to modify job state after deployment, and difficulty diagnosing data correctness issues.

To solve these problems, an external graph‑service (editable execution plan) is introduced. It statically analyzes job topology, extracts per‑operator TTL, and exposes the configuration for user editing. Modified TTL values are injected back into the TableConfig before the job is submitted, enabling fine‑grained resource control.

Two core processes are implemented: TTL collection during the ExecNode‑to‑Transformation conversion, and execution‑plan enhancement that applies the edited TTL settings. This approach reduces container CPU usage from over 100% to under 15% and shrinks checkpoint size from 8.54 GB to 1.8 GB.

Beyond TTL, the editable plan also supports operator‑level changes such as adjusting parallelism, slot‑sharing groups, and chain strategies while preserving state. A comprehensive compatibility analysis classifies migration scenarios into COMPATIBLE_AS_IS, COMPATIBLE_AFTER_RENAME, COMPATIBLE_AFTER_MIGRATION, and INCOMPATIBLE, based on AST, topology, and state schema checks.

For state migration, a KeyedStateMetadata structure records each keyed state’s name, type, TTL, and a custom StateContext for schema compatibility. During migration, the old savepoint is read, transformed via the State‑Process‑API, and written to a new savepoint, enabling seamless recovery for added aggregations, joins, or deduplication.

A pre‑analysis capability validates SQL AST compatibility, topology consistency, and state schema before migration, preventing wasted resources and operational overhead.

To aid correctness debugging, a trace‑like system is built using Byte‑Buddy bytecode instrumentation. Critical methods in AbstractStreamOperator (e.g., setKeyContextElement, processWatermark) are intercepted to capture input/output records, which are then streamed to Kafka and stored in an OLAP engine for query‑based investigation.

The architecture consists of a platform entry point, TM‑side instrumentation, Kafka ingestion, OLAP storage, and a query UI. Data is emitted for all operators except sources and sinks, with optional operator selection. Two data‑export strategies are evaluated: log‑file forwarding (high loss risk) and socket‑based direct Kafka streaming (back‑pressure aware).

Three real‑world cases demonstrate the tool’s effectiveness: (1) uncovering a Flink bug in localtimestamp handling, (2) diagnosing nondeterministic ordering caused by MapState in joins, and (3) identifying state TTL misconfiguration leading to data loss. In each case, troubleshooting time dropped from days to minutes.

Future work includes extending fine‑grained resource management to SQL, making Flink SQL state queryable, lazy state migration, and enhancing the debugging tool with risk warnings and automated anomaly detection.