Data Quality Mastery: From Expectations to Operational Assurance

The story begins when a colleague from the business side asks, "Why do our reports often seem inaccurate?" To address this, the team presents a four‑step framework for data quality management.

01 About Data Quality Expectation

Understanding the data quality expectations of the requester is essential. Expectations are defined qualitatively and quantitatively, and must be clarified early in the demand‑review stage. The team proposes three groups of questions to capture expectations, assess risks, and verify business knowledge.

These questions help align expectations, reduce misunderstandings, and guide downstream developers and consumers.

02 About Data Quality Measurement

Measurement compares actual data against the defined rules. Rules are split into basic (platform‑provided) and customized rules derived from quality expectations. An example case uses a "Business Object Exposure and Click Log" to illustrate rule extraction.

Rules consist of indicators and judgments (e.g., "Transmission loss rate < threshold"). Measurement occurs at three stages: initialization, acceptance, and production, each with specific reporting requirements.

03 About Data Quality Assurance

When measurements reveal issues, targeted actions are taken. Using the same log example, a missing value in attribute f5 is traced to the reporting stage, leading to a client‑side fix and a new test case to prevent recurrence.

Four assurance categories are defined: process assurance, policy assurance, monitoring assurance, and resource assurance, each with concrete practices such as data admission workflows, change‑release procedures, responsibility assignments, and resource allocation.

04 About Data Quality Operation

Operationalizing quality involves detecting, intercepting, and handling anomalies at scale. The team establishes operational goals—reducing incident loss and improving assurance efficiency—and builds an indicator system to track governance objectives, strategies, and evaluations.

Standardization, rule design, and tool support (monitoring, baseline management, DQC tools) enable continuous improvement. Effective alert management, including classification of valid, invalid, unresponsive, escalated, false, and missing alerts, is demonstrated through a case where weekly audits reduced alert volume from over 2,000 to under 100 per week.

Overall, the framework provides a systematic approach to data quality from expectation setting through measurement, assurance, and operational governance.