Data Quality Monitoring: Standards, Practices, and Technical Solutions

With the rise of the big‑data era, many industries are focusing on data to empower business, and data quality has become the foundation for data warehouses, data mining, and data‑driven operations. Effective data‑quality monitoring requires a governance framework that includes metadata management, standardization, and service management.

1. Data Quality Evaluation Standards

Integrity : Ensures that records are complete and no fields are missing, preventing inaccurate statistics caused by unsynchronized schema changes.

Accuracy : Verifies that recorded information is correct and free of anomalies, often monitored through key business metrics such as daily active users or revenue.

Consistency : Checks that the same metric yields identical results across different systems, avoiding discrepancies caused by varying calculation logic.

Timeliness : Guarantees that data is produced promptly, so its value can be realized; this is measured by whether results are computed before a specified deadline.

2. Scope of Data Monitoring

Data monitoring includes performance monitoring (read/write latency, resource queues, node consumption), daily monitoring, and data reconciliation. The focus here is on daily monitoring and reconciliation, which help assess asset‑quality status, support platform optimization, and provide decision‑making data for resource allocation.

Key Monitoring Points :

Monitor data‑asset quality (e.g., synchronization consistency) to guide data‑warehouse performance improvements.

Implement proactive alerts for data‑quality issues, covering both offline and real‑time pipelines.

Standardize fault tracking, review, and continuous improvement processes.

Establish a closed‑loop workflow from technical detection to business supervision.

3. Technical Solutions

For a system with over 3,600 tables and 15,000 tasks, the following aspects are critical:

Identify core intermediate tables used by key business indicators (e.g., site selection, channel, campus) and set up alerting for any anomalies.

Monitor low‑level data changes, as even a single field modification can cause downstream failures.

Optimize task scheduling (e.g., Airflow) by splitting core tables into incremental syncs to reduce full‑sync pressure.

Weekly on‑call responsibilities include monitoring alarm groups, handling feedback, and tracking core‑table data fluctuations.

4. Challenges

• Alarm fatigue: Too many alerts lead to important warnings being ignored. Solutions include adding feedback tags to classify normal alerts and highlighting critical business alarms with distinct formatting.

• Ensuring accuracy across long data pipelines: Each processing stage must be monitored, and code reviews are required to guarantee logical correctness.

Data quality monitoring is an iterative process; continuous improvement and knowledge sharing are essential for sustaining business value.

Author : Xi Pei, Data Warehouse Specialist at TAL Education Group.