Field-Level Data Lineage Extraction for FlinkSQL Using Apache Calcite

Data lineage (data lineage) is a crucial component of data governance, enabling traceability of data from creation through processing to consumption; it helps developers quickly locate issues, track changes, and assess upstream/downstream impacts.

The article introduces Apache Calcite, an open‑source dynamic data management framework that provides a standard SQL parser, validator, optimizer, and connector capabilities without handling data storage, and explains its role as the SQL engine within Flink.

It then outlines the FlinkSQL execution flow, which consists of five stages—Parse, Validate, Convert, Optimize, and Execute—detailing how SQL is transformed into relational expressions (RelNode) and eventually into a physical execution plan.

The core lineage extraction method follows three steps: (1) parse the input SQL to generate a RelNode tree, (2) generate an optimized logical plan (stopping before physical planning), and (3) query the optimized RelNode using RelMetadataQuery.getColumnOrigins to retrieve original field origins and construct lineage relationships.

Implementation details show how the parseFieldLineage method creates the original RelNode, produces the optimized logical plan, and invokes Calcite’s metadata API to obtain lineage information.

Practical examples demonstrate Insert‑Select and Insert‑Join scenarios, illustrating how lineage graphs map source tables and fields to target tables, and highlight limitations with lookup joins, watermarks, UDTFs, and CEP that initially lack lineage support.

Extensions are presented to handle lookup joins by adding a getColumnOrigins method for snapshot RelNodes, and to support UDTFs by extracting field origins from lateral table calls, enabling full lineage capture for these advanced constructs.

Finally, the article discusses capturing transformation relationships (e.g., functions like date_format or custom UDFs) by further modifying Calcite’s metadata handling, allowing the lineage graph to reflect not only source‑target mappings but also the processing steps applied to fields.

The overall contribution provides a concise, extensible approach to field‑level data lineage in real‑time data warehouses built on FlinkSQL.