How to Build a Reliable Real‑Time Data Warehouse: Timeliness, Quality, and Cost Strategies

Timeliness Assurance

<code>1: Monitor Kafka latency: check Flink consumer lag (e.g., Kafka lag).
2: Balance layering and latency to ensure reusability without overly long pipelines.
3: Handle out‑of‑order data.
4: Conduct early stress testing for traffic spikes, especially during major promotions, to secure resources and optimize tasks.
5: Set latency baselines and keep them within limits by optimizing code, resources, and addressing skew and back‑pressure.
6: Monitor metrics such as task failover, checkpoint status, GC, and back‑pressure, and trigger alerts on anomalies.
7: Use Flink LatencyMarker to observe pipeline latency.</code>

Quality Considerations

The two mainstream real‑time processing architectures, Lambda and Kappa, each have strengths and weaknesses but both ensure eventual consistency, contributing to data quality. Combining their advantages into a hybrid architecture will be explored in a future article.

Data quality is a broad topic; merely supporting re‑runs and back‑fills does not fully resolve it. A dedicated discussion on big‑data quality issues will follow.

<code>## Data Consistency
1: Ensure end‑to‑end consistency in real‑time computation; use idempotent output where storage supports overwrites. For non‑idempotent stores like Kafka DWD, deduplicate downstream with row_number() to avoid double processing.
2: Align offline and real‑time consistency by using identical data sources and business logic.

## Data Completeness
Goal: Preserve effective data from source through processing to presentation, avoiding loss due to processing logic, permission issues, or storage failures.
Examples:
- Back‑pressure in the source (MQ/Kafka) can cause message pile‑up and eventual loss.
- Incorrect processing logic may drop required data.
- Storage capacity exhaustion prevents new data writes.
- Completeness includes processing correctness, timeliness, and rapid recovery.

## Data Processing Correctness
Transform raw source data into target metrics as required by business logic, e.g., filter irrelevant alliance clicks and enrich remaining clicks with account, plan, and unit information before aggregation.

## Data Processing Timeliness
Control the time from data generation to front‑end display within reasonable bounds.

## Data Rapid Recovery
When pipeline interruptions occur, resumed processing must quickly catch up without duplication or omission.
Examples:
- Resolve consumer performance issues to alleviate backlog.
- Restart after consumer bugs to resume normal consumption.

## Data Observability
Key nodes in the data flow should be monitorable.

## Data High Availability
Design both processing and storage clusters with redundancy and disaster‑recovery capabilities.</code>

Stability Considerations

<code>Task Stress Testing
- Perform early load tests for peak traffic, especially during large promotions, to ensure resource provisioning and task optimization.

Task Prioritization
- Define protection levels based on impact scope and data consumers; prioritize company‑wide tasks over departmental, and external over internal usage. High‑priority tasks receive immediate response and may use dual‑path safeguards.

Metric Monitoring
- Monitor failover, checkpoint, GC, and back‑pressure metrics, issuing alerts on anomalies.

High Availability (HA)
- Choose HA components throughout the real‑time pipeline, support data backup and replay, and enable dual‑run fusion for critical business paths.

SLA Assurance
- Under HA guarantees, support dynamic scaling and automatic workflow migration.

Elastic Antifragility
- Implement rule‑based and algorithmic elastic scaling; provide failure handling for event‑triggered action engines.

Monitoring & Alerts
- Provide multi‑layer monitoring (cluster, physical pipeline, logical data layer).

Automated Operations
- Capture and archive missing or erroneous data, with periodic automatic retries to fix issues.

Upstream Metadata Change Resilience
- Ensure upstream business schemas are compatible with metadata changes; real‑time pipelines must handle explicit field modifications.</code>

Cost Considerations

<code>Labor Cost
- Democratize data applications to reduce talent expenses.

Resource Cost
- Enable dynamic resource utilization to cut static waste.

Operations Cost
- Adopt automated operations, HA, and elastic antifragility to lower OPEX.

Trial‑and‑Error Cost
- Embrace agile development and rapid iteration to minimize experimentation expenses.</code>

Agility Considerations

<code>Agile big data represents a comprehensive theory and methodology; from a data‑usage perspective, agility means configurability, SQL‑centric workflows, and democratization.</code>

Management Considerations

Data management is extensive; this article focuses on two key aspects: metadata management and data security management. Unifying metadata and security across diverse modern data‑warehouse storage choices is challenging. The real‑time pipeline will provide built‑in support for both, while also allowing integration with external metadata and security platforms.

This article discusses common task‑guarantee methods and theories for building real‑time data warehouses; practical implementations may vary, but the overall concepts are worth referencing.