Fundamentals of Data Quality Management: Rules, Metrics, Profiling, and Cleaning

Introduction: Data quality assurance involves rules, metrics, data profiling, governance mechanisms, and cleaning, which are essential for anyone working on data quality.

Fundamentals: Data Quality Management (DQM) covers the entire data lifecycle, identifying, measuring, monitoring, and improving data quality across stages.

The six key dimensions of data quality are completeness, timeliness, validity, consistency, uniqueness, and accuracy.

Data quality rules and metrics: The article presents a comprehensive set of rules and indicators organized by object (single column, cross‑column, cross‑row, cross‑table, cross‑system) with example metrics such as null‑value rate, sample anomaly ratio, and foreign‑key consistency.

对象

质量特性

规则类型

指标

单列

完整性

不可为空类

空值率

有效性

语法约束类

1-样本记录异常值比率

有效性

格式规范类

有效性

长度约束类

有效性

值域约束类

准确性

事实参照标准类

样本记录中真实记录的比率

跨列

完整性

应为空值类

及时性

入库及时类

满足时间要求的样本记录的比率

一致性

单表等值一致约束类

一致性

单表逻辑一致约束类

跨行

唯一性

记录唯一类

一致性

层级结构一致约束

跨表

一致性

外关联约束类

外键无对应主键的样本记录比率

一致性

跨表等值一致约束类

一致性

跨表逻辑一致约束类

跨系统

一致性

跨系统记录一致约束类

样本记录与其它系统的匹配率

及时性

入库及时类

满足时间要求的样本记录的比率

Data profiling: Profiling is a crucial step that helps detect common issues such as missing values, duplicate keys, value‑range anomalies, and logical inconsistencies; typical profiling items and their analytical meanings are listed in a table.

Data quality assurance mechanism: Continuous improvement relies on automated, regular monitoring, including designing quantitative indicators, scoring rules, assessment, anomaly monitoring, visualization, and alerting responsible owners.

Data cleaning: Data cleaning (re‑validation) removes duplicates, corrects errors, and ensures consistency, serving as a key process for improving existing data quality when upstream controls are insufficient.

Conclusion: The author invites readers to follow, share, and join discussions on data governance, offering templates and further resources via a public account.