Real-Time Data Warehouse: Background, Value Assessment, and Half-Year Progress

Data has become an indispensable production factor, and the data middle platform (data warehouse) aims to provide efficient, high‑quality services across the company. This report introduces the background, terminology, and a brief overview of the real‑time data warehouse team's initiatives over the past six months.

1. Background and Terminology Key terms include Data Warehouse (数仓), Minute‑Level Warehouse (分钟级数仓) as a real‑time solution, the "Three‑High One‑Low" principle (high efficiency, high quality, high stability, low cost), and the internal BI platform TianShu (天枢).

2. Data Warehouse Value Thinking Assuming production costs are not constrained, the value can be expressed as: Data Warehouse Value = Indicator Unit Value × Number of Indicators × Average Production Efficiency × (1 - Indicator Acquisition Cost Coefficient) × PV × UV From this formula, the team derives actions such as prioritizing high‑value demands, enriching domain indicators, accelerating indicator production, ensuring timely delivery and understandability, increasing exposure, and expanding user base. These lead to two main value streams: data production (value creation) and data operation (value amplification).

3. Half‑Year Work of the Real‑Time Data Warehouse

3.1 Establishing Minute‑Level Warehouse as the Overall Architecture The minute‑level warehouse was chosen for its high availability and performance, and key scenarios like continuous reporting, conversion, live courses, and real‑time sales have been migrated successfully.

3.2 Designing a Quality Assurance Scheme To address unstable job chains, a comprehensive monitoring system was built, covering basic monitoring (ODS consistency, batch generation, data update times), advanced monitoring (key indicator tracking, offline‑real‑time fallback), and automated coverage reaching 100%.

3.3 Pioneering Real‑Time Data Governance Guided by the "Three‑High One‑Low" principle, the team applied OSM and PDCA methodologies, defined stability metrics (SLA‑bound incidents, resolution time), and identified factors affecting stability from both subject (code, scripts) and object (tools, resources, engines) perspectives.

3.4 Expanding Data Value Efforts include multiple data delivery channels (BI, Kafka, T‑query, T‑service), establishing minute‑level development standards, and building a data dictionary to improve data accessibility and consistency.

3.5 Externalizing Data Warehouse Value The team visualized data assets, tracked production and operation outcomes (e.g., reduced break‑line rate, decreased S0 incidents, improved monitoring coverage), and showcased stability improvements through dashboards.

4. Conclusion A complete minute‑level data warehouse solution has been delivered, ensuring stable operation and providing a methodology that can be quickly replicated for other teams, while ongoing improvements continue to enhance efficiency, quality, stability, and cost.