Building Real-Time Data Warehouses with Apache Flink: Goals, Architecture, and Best Practices

The article, based on a series of Apache Flink live sessions and shared by Meituan's data systems engineer Huang Weilen, explains why real‑time data warehouses are needed: to address the low timeliness of traditional warehouses and to solve problems that require up‑to‑date data.

Two core principles are proposed: (1) real‑time warehouses should not duplicate the functions of offline warehouses, and (2) they should not be used for tasks better solved by other systems, such as highly time‑sensitive or heavily business‑driven logic.

Typical use cases include real‑time OLAP analysis, live dashboards, real‑time feature generation, and business monitoring, with examples from Meituan and e‑commerce events.

The architecture mapping compares offline and real‑time layers, showing how Hive SQL maps to Flink SQL, MapReduce/Spark jobs map to continuous Flink streaming jobs, and how storage shifts from HDFS to Kafka, with dimension data often stored in KV stores like HBase.

Layer‑wise construction is detailed:

ODS layer: unified real‑time sources (Kafka, binlog, logs) with partition‑level ordering.

DW layer: data cleaning, versioning, unique keys, and handling of duplicate or out‑of‑order records.

Dimension handling: low‑frequency dimensions are cached, high‑frequency dimensions use changelog (link) tables, often backed by HBase.

Aggregation layer: unified metric calculations, use of approximate algorithms (Bloom filter, HyperLogLog) for distinct counts, and careful window and TTL configuration.

Quality assurance involves a toolchain that provides job submission, resource allocation, monitoring, metadata management, and lineage tracking. Metadata is loaded into Flink catalogs, DDL parsing updates table definitions, and job status is recorded to maintain accurate data lineage.

Data validation is performed by writing real‑time results to Hive and comparing them with offline pipelines, enabling automated alerts when discrepancies exceed thresholds.

Overall, the article offers practical insights and best practices for building, operating, and maintaining a high‑quality real‑time data warehouse using Flink.