How to Master Data Quality Management in the Big Data Era

Data Quality Concept

In the digital age, big data is a strategic resource, but its characteristics—distributed storage, high computational complexity, and high‑value relational presentation—make guaranteeing data quality difficult, yet data quality is decisive for effective data use and decision‑making.

Root Causes of Data Quality Issues

Based on extensive practice, ten common causes are identified across people, processes, technology, and information:

1. Data multiplicity : Multiple sources produce inconsistent values, often unnoticed due to independent data production processes.

2. Subjective judgment in data generation : Human bias can be embedded in collected facts.

3. Limited computing resources : Insufficient resources restrict data accessibility.

4. Trade‑off between security and accessibility : Balancing privacy protection with the need for high‑quality data access creates conflict.

5. Cross‑disciplinary data encoding : Lack of interoperable codes hampers comprehensive data collection.

6. Complex data representation : Unstructured text or image data are hard to analyze and summarize.

7. Excessive data volume : Large volumes impede timely data retrieval.

8. Overly strict or ignored input rules : Rigid or neglected validation rules cause data loss or errors.

9. Changing data requirements : Shifts in business context alter what constitutes useful data.

10. Distributed heterogeneous systems : Inconsistent definitions, formats, and rules across systems hinder integration.

Data Quality Management System

The system consists of five layers, with the top layer defining the data quality management strategy (vision and principles). It aligns organizational structure, roles, responsibilities, and processes with enterprise strategy, and integrates with data security management to balance accessibility and protection.

The management process starts from business pain points, conducts data profiling, root‑cause analysis, and formulates systematic solutions that are monitored in daily operations to achieve continuous improvement.

Data Quality Evaluation Methods

Evaluation follows three steps: (1) collect stakeholder expectations via interviews or questionnaires and measure quality with multidimensional indicators; (2) compare subjective and objective assessments to identify gaps and their causes; (3) communicate findings, define improvement plans, and implement actions.

Key Evaluation Dimensions

Accuracy : Measures correctness of data, often derived from rule‑based error detection.

Completeness : Assesses architectural, attribute, and dataset completeness.

Consistency : Evaluates referential, element‑level, and format consistency.

Conformity : Checks alignment with standards, models, business rules, and reference data.

Accessibility : Gauges the effort and time required to obtain data.

Timeliness : Reflects data freshness, often measured by data age.

Future Outlook

Data quality improvement is an ongoing journey; evolving business environments continuously introduce new challenges and opportunities. Advances in data mining, cleaning, and preprocessing technologies, especially within big‑data platforms, are expected to enhance quality assessment and remediation capabilities.