Data Quality Management: A Complete Guide to Dimensions, Processes, and Roles

What Is Data Quality

Data quality refers to data that conforms to its intended purpose. High-quality data accurately reflects real-world conditions. Data quality management is a set of practices aimed at maintaining high-quality information throughout the data lifecycle — from collection and advanced data processes to effective distribution — with management oversight. Effective data quality management is essential for consistent data analysis because data quality is critical for obtaining actionable and accurate insights.

Importance of Data Quality

Most corporate operations and strategic decisions now rely heavily on data, raising the stakes for quality. Poor data quality is the primary cause of failure in advanced data and technology initiatives. It affects productivity, the bottom line, and overall ROI. Benefits of effective data quality management ripple across CRM, supply chain, and ERP systems. With quality data, organizations can build data warehouses to examine trends and formulate forward-looking strategies. According to an Accenture big data survey, 92% of executives using big data for management were satisfied with results, and 89% considered data "very" or "extremely" important, believing it will revolutionize operating models like the internet did.

Key Benefits

Better Decisions: Consumer-centric markets demand high-quality data for better decision-making.

Better Team Collaboration: Consistent access to the same high-quality data across departments improves communication and alignment on priorities, external messaging, and brand.

Better Customer Understanding: Quality data enables accurate assessment of customer interests and needs, driving growth through better products and marketing campaigns based on direct feedback rather than guesses.

Improved Business Processes: Good data helps teams identify failure points in operational workflows, especially in supply chains that rely on real-time data for inventory and shipment tracking.

Data Quality Assessment Dimensions

Data quality is evaluated across multiple dimensions that may vary by source. The article adopts six dimensions as key indicators:

Completeness: Measures what data is missing or unavailable.

Conformity: Measures data not stored in a unified format.

Consistency: Measures conflicting values in terms of information meaning.

Accuracy: Measures incorrect or outdated data.

Uniqueness: Measures duplicate data or duplicate attributes.

Integration: Measures missing or unindexed related data.

An illustration of these six indicators is provided in the article.

Sources of Data Quality Problems

Mergers & Acquisitions: Combining companies often means merging disparate data systems, legacy databases, different collection methods, or inheriting incorrect data.

Legacy System Transitions: Switching operating systems is inherently difficult; data migration rarely happens seamlessly.

User Errors: Humans will always be involved in data entry and will make mistakes (typos, misentries).

Common Data Quality Issues (12 Types)

Lack of Record Uniqueness: Organizations use ~123 SaaS apps; multiple records for the same entity are common.

Lack of Relationship Constraints: Undefined relationships between datasets lead to incorrect and incomplete information.

Lack of Referential Integrity: Example: Sales records referencing non-existent Product IDs in the Product table.

Lack of Relationship Cardinality: Maximum relationship counts between entities (e.g., one customer–one location, one customer–many purchases) are not enforced.

Lack of Attribute Uniqueness and Meaning: Duplicate column names with different info, different column names meaning the same thing, ambiguous column headers.

Lack of Validation Constraints: Missing checks cause extra spaces, inappropriate symbols, excessive length in fields like CustomerName.

Lack of Accurate Formulas and Calculations: Derived fields (e.g., profit margin) computed from erroneous formulas propagate errors across entire columns.

Lack of Cross-Source Consistency: Same entity (e.g., customer) stored in CRM and billing systems diverges over time.

Lack of Data Completeness: Missing required fields; completeness measured vertically (attribute-level) or horizontally (record-level). Fields must be classified as mandatory (e.g., customer name) or optional (e.g., hobby).

Lack of Data Currency: Data ages quickly (address, email, phone changes); outdated data leads to decisions based on stale information.

Lack of Data Literacy Skills: Employees store wrong information because they don't understand attribute meanings or downstream impacts of their updates.

Typos and Human Errors: Humans make ~400 errors per 10,000 entries; even with unique IDs, validation, and integrity constraints, errors persist.

Common Data Quality Monitoring Metrics (14 Items)

Incoming data volume fluctuation (7-day moving average comparison)

Source system schema changes (table names, column names, column types)

Source table computation latency causing downstream ingestion delays

New values in online dimension tables not yet updated in the warehouse

Primary key and foreign key checks on ingested RDBMS tables (duplication)

Important field length checks

Null value checks

Important field enumeration distribution checks (discrete, mostly dimension fields, can include null checks)

Value range checks (continuous, usually fact fields: MAX, MIN, SUM, AVG)

Date validity checks (cross-day logs, abnormal timestamps)

Data-to-error ratio (known errors vs. entire dataset)

Null count (frequency of empty fields)

Time to data value (time to derive insights)

Data transformation error rate (frequency of failed transformation operations)

Data storage cost (rising cost with flat or decreasing usage indicates low-quality stored data)

Which Companies Should Prioritize Data Quality

Cost-focused organizations: Retailers, oil & gas firms, government agencies, healthcare providers — cutting waste from bad data is better than indiscriminate layoffs.

Data-driven organizations: Companies selling/licensing data, monetizing data, deploying analytics widely, experimenting with AI, digitizing operations.

Organizations with unclear data ownership: Business blames IT, IT blames business; senior management must assign primary data responsibility to business units.

Decision-makers distrusting their data: Better data yields better decisions, less stress, and more time for complex decisions.

Roles and Responsibilities in Data Quality Management

1. Data Quality Manager

Collaborate with stakeholders to define requirements.

Analyze data conformance to requirements.

Create metrics to track progress.

Implement new policies/processes to improve quality.

Monitor progress over time via metrics.

2. Chief Data Officer (CDO)

C-level executive accountable for data assets.

Establish organizational goals for data quality.

Formulate data usage and control policies.

Oversee policy implementation and compliance mechanisms.

Prioritize data quality projects.

Integrate data quality across departments.

Train staff on best practices.

Advocate for improved data practices internally and externally.

Supervise data quality processes to ensure business requirements are met.

Develop strategies for using data to achieve business goals.

3. Data Steward

Defines data usage and security policies per governance strategy; allocates resources for database maintenance; ensures policy compliance; monitors and reports quality. Acts as gatekeeper, leading project planning, report reviews, development meetings, process design, and change advocacy. Negotiates cross-functional differences among stakeholders.

4. Data Analyst

Collect data from various sources.

Analyze collected data.

Design and maintain data systems and databases.

Make predictions based on findings.

Communicate clearly with cross-department colleagues.

Collaborate with programmers, engineers, and executives to enhance processes, modify systems, and build governance strategies.

Work closely with data scientists who focus on predictive modeling.

5. Big Data Engineer

IT professional using big data technologies to analyze large datasets; designs, builds, tests, maintains, monitors, and manages complex data infrastructure (relational databases, columnar databases, distributed file systems, caching algorithms, information retrieval methods).

6. Data Architect

Designs company data architecture: gathers business requirements, analyzes current structures, plans future architecture. Strategic thinker who understands how technology changes affect the data environment; develops all technical aspects of data architecture; aligns with organizational initiatives; manages IT partner/vendor relationships.

Data Quality Management Process

The ultimate goal: enhance data's value in use through reliable data, ultimately delivering economic benefits to the enterprise. Data quality management is a core component of data governance, applied in data modeling, data asset management, master data management, and data warehouse solutions. It can be reactive (passive) or preventive (active); combining technical controls with business processes improves proactive capability.

Guiding Principle: "Prevention First, Integration of Prevention and Control, Continuous Optimization"

Management spans the data lifecycle, strengthening pre-event prevention, in-process control, and post-event remediation for continuous improvement.

1. Pre-Event Prevention

Organizational Construction

Role Definition: Establish data quality roles within the overall data governance framework with clear responsibilities.

Personnel Training: Human factors are the main cause of inaccuracies; training raises awareness and reduces issues.

Data Standards Enforcement

Effective execution of data standards (model standards, master/reference data standards, metric standards) is a prerequisite.

Policy and Process Assurance

Data Quality Management Closed-Loop Process

Includes: Business Requirements Definition → Data Quality Measurement → Root Cause Analysis → Improvement Implementation → Quality Control.

Business Requirements Definition: Align business goals with quality strategy; involve business users deeply; clearly define business problems to trace root causes.

Data Quality Measurement: Design assessment dimensions and metrics around business needs; use tools to evaluate data sources; categorize issues and analyze causes. Combine top-down (business-goal-driven) and bottom-up (data profiling) approaches; produce governance assessment report.

Root Cause Analysis: Distinguish symptoms from root causes; set quality control points at the source.

Improvement Implementation: No universal solution; define rules and metrics based on root causes and business impact; create a unique improvement plan and execute immediately.

Quality Control: Erect a data quality "firewall" — automated measurement and monitoring programs at problem entry points based on root cause analysis and handling strategies — to prevent bad data from propagating downstream.

Data Quality Management Policies

Set KPIs and scoring for business domains/departments; quantify issues by category weight; summarize patterns; use tools for regular monitoring and measurement; drive timely corrections. Policies constrain parties to strengthen awareness and proactive problem-solving.

2. In-Process Control

Monitor and manage quality during data maintenance and usage. Establish a process-oriented control system covering creation, change, collection, cleansing, transformation, loading, and analysis.

Strengthen Source Control

Maintain Data Dictionary: Critical tool for recording standard data and ensuring quality; eliminates cross-department misunderstandings; saves IT project time and cost.

Automate Data Entry: Reduce human error (e.g., auto-match customer info by keyword and populate forms).

Automate Data Validation: Pre-set quality rules to validate on entry; alert or reject non-conforming data. Rules include: data type correctness, deduplication, domain value ranges, classification rules, unit correctness.

Manual Review: Process-driven model where every addition/change requires human approval (e.g., supplier master data changes).

Strengthen Flow Control

Data Collection: Define requirements, standardize process/model, ensure accurate/timely/complete source data, broadcast changes via messages, ensure granularity meets business needs, define acceptable value ranges, validate tools/methods/processes.

Data Storage: Choose appropriate DB systems, design rational tables, store at proper granularity, establish retention schedules, define ownership and access rights, set access/query guidelines.

Data Transmission: Define boundaries/limits, ensure timeliness/completeness/security, guarantee reliability against tampering, assess tech/tool impact on quality.

Data Processing: Process rationally per business goals; handle duplicates, missing values, outliers, inconsistencies.

Data Analysis: Ensure algorithms/formulas/systems are valid and accurate; ensure input data complete and valid; analyze reproducibly; use appropriate granularity; show proper comparisons and relationships.

Ongoing In-Process Strategies

Continuous Rule Updates: Regularly check rule relevance to business; improve rules; add new rules for changing environments.

Continuous Monitoring: DQAF's "online measurement" — automate defined rules on data sources for continuous validity checks to detect issues and determine improvements.

Advanced Technology (AI): Use AI for better data identification/parsing, quality quantification, issue analysis, matching/deduplication, and data enrichment.

Alerting Mechanism: Notify data stewards via WeChat, SMS, email with issue lists for timely remediation.

Quality Reporting: Dashboards showing KPIs for analysis/location; issue logs recording known problems to prevent recurrence.

3. Post-Event Remediation

Prevention and control cannot eliminate all issues; "slippage" always occurs. Human intervention always introduces some errors; even automated processes can fail. Timely detection and remediation minimize impact.

Periodic Quality Monitoring

Regular re-evaluation of non-critical or non-continuously-measured data (like periodic health check-ups) to assure conformance, detect changes, and formulate improvements.

Data Issue Remediation

Clean Duplicate Data: Delete or merge (keep most complete/accurate record).

Clean Derived Data: Store algorithms/formulas instead of results (e.g., profit margin) to avoid storage/maintenance cost and re-calculation errors when logic changes.

Handle Missing Values: Impute via manual (small data, master data) or automatic (big data) methods.

Handle Outliers: Core is detection; many methods use machine learning.

Continuous Improvement and Optimization

Data quality management is a virtuous cycle: measure → analyze → investigate → improve. Through continuous strategy optimization, shift from reactive response to proactive prevention. After measurement, root cause analysis, and repair, reassess: data model design rationality, process standardization (creation, change, collection, storage, transmission, processing, analysis), rule/threshold appropriateness. Implement optimizations where gaps exist. Post-remediation is never ideal; adhere to prevention-first, using continuous measurement and investigation to perpetually discover issues, refine methods, and elevate quality.

Conclusion

Data quality affects not only IT project success but also business collaboration, management innovation, and decision support. Adhering to "garbage in, garbage out" and the "prevention, control, remediation" strategy continuously elevates enterprise data quality. While no foolproof method prevents all issues, embedding data quality into the organizational DNA largely earns trust from business users and leaders. As big data evolves and data demands grow, solving quality issues is more critical than ever. Technology advances, business changes, and data volume increases make environments increasingly complex. Data quality management is a continuous process — there is no "perfect timing"; the best time to start is now.