How Meituan Optimized Flink SQL: Fine‑Grained Config, State Migration, and Debugging

01 Flink SQL in Meituan

Meituan has more than 100 business units using Flink SQL, with over 5,000 SQL jobs accounting for 35% of all Flink workloads and a 115% year‑over‑year growth rate.

02 SQL Job Fine‑Grained Configuration

Flink does not support fine‑grained TTL, partitioning, or parallelism settings for SQL jobs. TTL can only be set at the job level, leading to resource waste. Two business scenarios illustrate the problem:

Different operators require different state TTLs (e.g., a deduplication operator needs 1 hour, while an aggregation operator needs 1 day).

Join operators on streams with mismatched business cycles and dimension tables require separate hot‑cold data handling.

Meituan introduced an external graph‑service (editable execution plan) that statically analyzes job topology, extracts TTL per operator, and exposes it for user editing. The edited TTL is passed to the Flink engine via TableConfig, enabling per‑operator TTL configuration.

Experiments show that fine‑grained TTL reduces peak container CPU usage from 107% to 14.8% and cuts checkpoint size from 8.54 GB to 1.8 GB.

03 SQL Job Change Support from State Recovery

Flink SQL’s native state recovery is strict; many job changes cannot resume from existing state. Meituan classified migration scenarios into Graph Migration, Operator Migration, and Savepoint Migration, focusing on Operator Migration for real‑time warehouse use cases.

Key steps:

Define KeyedStateMetadata to describe each keyed state (name, type, TTL, compatibility context).

During job upgrade, collect metadata, read the old savepoint, transform state via the State‑Process‑API, and write a new savepoint.

Three‑layer compatibility checks (SQL AST, editable execution plan topology, and state schema) produce four outcomes: COMPATIBLE_AS_IS, COMPATIBLE_AFTER_RENAME, COMPATIBLE_AFTER_MIGRATION, and INCOMPATIBLE.

04 SQL Correctness Issue Investigation

Meituan built a Flink SQL debugging system to trace data through operators, inspired by distributed tracing but adapted to Flink’s operator model. Byte‑Buddy bytecode instrumentation captures key methods ( setKeyContextElement, processWatermark) to record input/output records.

Data flow:

Instrumentation extracts serialized RowData (or HoodieRecord via toString) and field metadata during TranslateToPlan.

Records are sent to Kafka, then ingested into an OLAP engine for query.

Three case studies demonstrate the tool’s value:

Bug in Flink’s localtimestamp handling caused data loss at exact‑second timestamps.

Design limitation: MapState in joins leads to nondeterministic ordering.

User misconfiguration of State TTL resulted in expired state and missing records.

Using the debugger reduced troubleshooting time from days to minutes.

05 Future Outlook

Planned enhancements include:

Fine‑grained resource management for Flink SQL via API and autopilot integration.

Queryable Flink SQL state and lazy migration support.

Risk‑alerting before job deployment based on accumulated debugging insights.

Resolution of known ordering and performance issues.