How Flink Table Store Powers Real‑Time Financial Data Warehousing

In the 2022 Flink Challenge, a real‑time data‑warehouse design for the financial sector was created to meet the growing demand for timely reports in banking.

Financial Real‑Time Warehouse Situation

Banking uses dimensional modeling with fact tables (transaction records) and dimension tables (e.g., account balances) that require frequent updates.

Typical Transaction Scenarios

Fact table: insert‑heavy, records each transaction event and supports incremental calculations such as deposits, withdrawals, and loan purchases.

Dimension table: update‑heavy, stores relatively static attributes like customer ID, phone, and especially account balance, which must be refreshed on every transaction for real‑time reporting.

Solution Options

Three approaches were evaluated:

Kafka‑based ETL : Handles incremental fact‑table calculations but cannot efficiently perform full‑table dimension updates and suffers from limited data persistence.

Micro‑batch ELT : Loads data in real time then runs periodic full‑batch jobs; this introduces latency, instability, and poor timeliness for dimension‑driven analytics.

View‑based ELT : Uses MPP databases (e.g., StarRocks) with nested views to compute on dimension tables; while it supports full‑table calculations, query cost is high for high‑frequency workloads.

Flink Table Store for Streaming Warehouse

Released in 2022, Flink Table Store provides a lake‑storage format that separates compute and storage, supports fast data ingestion, rapid updates, and both batch and streaming reads.

Lake storage with dual‑write to data files and a log system.

Supports stream‑batch writes and fast update operations.

Integrates with Hive, Trino, StarRocks and other OLAP engines.

Data Ingestion

Full data from the Oracle‑based banking system is loaded via a custom JDBC connector with changelog‑producer = input. Incremental changes are captured by CDC tools, serialized as JSON, and sent to Kafka for downstream consumption.

Initialize customer balances (e.g., 100, 200, 300).

Process updates: a deposit adds to the balance, a withdrawal subtracts from it, each generating an Update in the dimension table.

CDC writes changelog JSON to Kafka; Flink SQL consumes the stream and upserts into Flink Table Store.

Configuration table.exec.sink.upsert-materialize = none preserves the complete changelog for later streaming reads.

Querying

Two query modes are supported:

Batch mode : Reads snapshots for historical data, enabling OLAP analysis via Hive, Trino, or Flink SQL.

Streaming mode : Continuously processes the changelog, providing low‑latency updates, unified SQL, mini‑batch optimization, and reduced materialization.

Exporting Results

For low‑frequency queries, Flink 1.16 SQL Gateway can be used directly; for high‑frequency access, results are streamed to an external database to provide stable online services.

Future Direction

The ultimate goal is an end‑to‑end streaming warehouse where data changes drive analysis, full changelog is stored in Table Store, and batch and streaming share identical SQL and storage layers.

In production, raw data is written to Flink Table Store, aggregation logic runs in Flink SQL, and summarized results are written to StarRocks, delivering near‑real‑time corporate CRM reporting for deposits and loans across branches.

For more technical details, see the Flink Table Store documentation at https://nightlies.apache.org/flink/flink-table-store-docs-release-0.2/.