This article explains how companies can transform collected data into valuable assets by addressing data assetization challenges, building comprehensive data platform functions, and developing team capabilities across governance, quality, security, and operational domains.
This article describes Bilibili’s data quality platform, outlining its background, objectives, theoretical models, workflow stages (recording, checking, alerting), DSL for metrics, root‑cause analysis, scheduling strategies, heterogeneous source integration, rule coverage, intelligent monitoring, and future plans to achieve automated, real‑time, high‑reliability data assurance for massive daily workloads.
Kuaikan Manhua ensures continuous data governance by establishing a closed‑loop of business scope management, data asset standards, and feedback mechanisms that keep data pollution slower than governance speed, enabling systematic, long‑term data quality improvement.
This article outlines a comprehensive data quality management framework, covering expectations, measurement, assurance, and operational practices, and provides concrete templates, rule designs, and governance processes to help data teams systematically assess, monitor, and improve data reliability throughout the lifecycle.
The article outlines a complete data‑quality‑management framework that first captures business expectations, then translates them into basic and personalized measurement rules, defines four assurance approaches for handling violations, and scales operation with indicators, tooling, and metrics to continuously improve data quality across the lifecycle.
This comprehensive guide explains how metadata connects source data, warehouses, and applications, outlines its technical and business classifications, demonstrates its value for data management, profiling, portals, and ETL development, and details optimization, storage, lifecycle, and quality practices essential for robust big‑data operations.
This article explains data governance fundamentals, outlines common evaluation shortcomings, and introduces observability concepts and the GuanCe Cloud platform as a way to objectively measure and improve governance outcomes across the entire data lifecycle.
This article explains what data governance is, why enterprises need it, the key components such as data quality, metadata, master data, asset and security management, and provides a step‑by‑step framework, organizational structure, platform features, evaluation methods and common pitfalls.
This article explains why data quality is critical for business success, outlines common data quality problems and their root causes, and presents a practical governance framework with monitoring rules, alerts, full‑link monitoring, and a seven‑dimensional evaluation model to continuously improve data reliability.
This article explains how Bilibili’s data quality team designs a process‑driven, automated, and AI‑enhanced platform that monitors billions of records daily, defines quality metrics such as completeness and consistency, integrates heterogeneous data sources, and provides root‑cause analysis and real‑time alerting to ensure trustworthy data for its massive user base.
The article explains why establishing simple, automated, and well‑governed data lifecycle processes—including CRUD, quality standards, and regulatory compliance—is essential for a result‑driven enterprise data strategy and outlines practical steps for designing such processes.
Bilibili built a scalable data‑quality platform that records metrics from heterogeneous sources, checks them with a rich DSL, alerts once with root‑cause analysis, and uses event‑driven and time‑window scheduling, automated workflows, and intelligent monitoring to ensure real‑time, accurate, trustworthy data for petabyte‑scale processing.
Amid the explosive growth of big data, this article explains why systematic data governance—covering metadata, quality, lifecycle, and security—is essential for turning raw data into measurable business assets, reducing costs, and enhancing operational efficiency.
This article details Kuaishou's end‑to‑end big data platform, describing its organizational model, unified data governance framework, comprehensive data‑quality solution, the design of a headless metric platform, key technologies such as automatic modeling and code generation, and future directions toward a decentralized, smart data fabric.
This article explains how X‑Select’s Data Quality Platform (DQC) addresses common data quality problems in large‑scale data development by defining six quality dimensions, leveraging open‑source solutions such as Apache Griffin and Qualitis, and implementing rule definition, execution, alerting, and workflow interruption within a Spark‑based architecture.
This article explains what objects and tags are, distinguishes physical, network, and electronic tags, outlines the structure of tag taxonomy, describes its applications in DMP, CDP, recommendation and user profiling systems, and presents construction principles and quality evaluation criteria for tag systems.
This article presents KuaiKan's data governance journey, detailing the rapid business expansion challenges, the three‑step planning framework, the logical closed‑loop architecture, practical implementation experiences, cross‑team collaboration techniques, and the evaluation of governance outcomes and future plans.
The Apache DolphinScheduler 3.0.0 release on August 10, 2022 introduces a faster UI, stronger data‑quality guarantees, modernized design, easier maintenance, AWS support, service splitting, and native Flink task support, accompanied by detailed code examples and download links.
This article presents a comprehensive overview of Du Xiaoman's risk control system evolution—from early rule‑based engines to AI‑enhanced intelligent decision engines—detailing technical practices such as strategy iteration acceleration, decision latency reduction, parallel workflow design, and future trends in data quality, automated strategy optimization, and real‑time analytics.
The talk outlines KuaiKan's data governance journey, describing the rapid business growth challenges, the three‑step logical closed‑loop framework, practical experiences in business scope management, data asset governance, collaboration techniques, and future outlook, highlighting evaluation metrics and ongoing improvements.
On August 10, 2022 Apache DolphinScheduler 3.0.0 was officially released, introducing a brand‑new Vue3‑based UI that is dozens of times faster, extensive AWS support, custom time‑zone handling, task groups, native data‑quality checks, service splitting for container‑native deployment, numerous new task types, Python API enhancements, and a long list of bug fixes and documentation updates.
In the era of big data, poor operations data quality undermines analytics, decision‑making and digital transformation, so organizations must adopt a three‑dimensional governance approach—covering organization, processes and technology—to ensure completeness, consistency, accuracy, uniqueness, relevance and timeliness of their operational data.
This article explains why event‑tracking data governance is essential, outlines the methodology and practical steps, and introduces a one‑stop platform that standardizes data collection, validation, monitoring, and visualization to improve data quality, efficiency, and business insight.
The author shares a candid account of transitioning from a non‑data role to confronting data‑centric bottlenecks, describing the current state of data projects, common pitfalls, and practical thoughts on simplifying data governance within limited resources and budget constraints.
The article presents an Indicator Testing Platform that automates metric validation—covering timeliness, completeness, accuracy, and consistency—through model‑level comparison, regression, online monitoring, and TDD‑style testing, dramatically reducing manual effort and enabling rapid detection and correction of data quality issues across thousands of business indicators.
The talk explains how to maximize the value of training data by exploring background on model generalization, automatic data augmentation techniques, denoising strategies, active learning for selecting unlabeled samples, and robust data splitting methods, offering practical guidelines for AI practitioners.
The article describes how combining Apache Flink with Hudi enables an incremental data lake that delivers near‑real‑time analytics by switching to merge‑on‑read, fixing log handling bugs, improving compaction planning, and refactoring table‑service scheduling, while showcasing use cases such as CDC ingestion, data quality control, and real‑time materialized views, and outlines future enhancements like optimistic concurrency and unified schema evolution.
This article presents a comprehensive case study of Zhihu's data empowerment team, detailing the design of a real‑time data platform and user profiling system, the challenges faced in scalability, latency, and data quality, and the practical solutions and architectural choices implemented to drive business value.
The article presents NetEase's internal data governance experience, outlining past challenges, current pain points, a comprehensive product strategy covering scope, value quantification, and feature implementation, and shares initial results and future plans to build an automated, end‑to‑end big‑data optimization platform.
The article outlines five major 2022 data trends— the rise of analytics engineers, the intensifying lake‑house competition, the growth of real‑time streaming pipelines and operational analytics, the expanding cloud marketplaces for data tools, and the push toward unified data‑quality terminology—explaining their origins, market impact, and future outlook.
The article details how to add field‑lineage tracking to Spark SQL by creating a custom SparkSessionExtension that injects a check‑analysis rule and a parser, which capture INSERT statements, analyze the physical plan, and generate a JSON mapping of source‑to‑target fields for data governance.
This article explains the background and importance of data governance, defines data quality and its key metrics, describes the monitoring and alerting modules of the IData data platform, outlines built‑in and custom rule templates, and discusses baseline management and future enhancements.
This tutorial demonstrates how to build a complete streaming data pipeline by configuring JDK, MySQL, Hadoop, Hive, Spark, Kafka, and Griffin, generating test data with shell scripts, processing it with Flink, and validating data quality using Apache Griffin in a Spark‑based deployment.
This article explains the challenges of early real‑time data pipelines, introduces a layered real‑time warehouse architecture, provides step‑by‑step Flink SQL code for building a demo warehouse, and covers comprehensive data governance, quality metrics, lifecycle management, and naming conventions for production‑grade big‑data systems.
This article introduces Apache Griffin, a model‑driven big‑data data‑quality monitoring platform, explains its key features, architecture, installation requirements, and provides step‑by‑step usage examples with Hive, Kafka and Spark integration.
In 2022 the modern data stack will be driven by the rise of analytics engineers, intensified competition between lakehouse and warehouse solutions, growing demand for real‑time analytics, the explosive growth of cloud marketplaces, and the emergence of unified data‑quality terminology, all reshaping data infrastructure and operational practices.
This article presents a comprehensive design of a real-time advertising data warehouse powered by Flink, covering construction background, technical and data‑warehouse architecture, real‑time OLAP, stability and data‑quality guarantees, future plans, and the integration of Hologres for simplified processing.
This article outlines a comprehensive data quality assurance framework for a massive reporting platform, covering the data pipeline architecture, detailed testing methods for timeliness, completeness, and accuracy, as well as application‑level checks, downgrade and backup strategies, and future automation plans.
This article presents MobTech’s comprehensive data governance and security practices, covering the necessity of governance, its benefits, a full‑chain governance framework, specific challenges in the financial sector, the evolution of their integrated architecture, and detailed implementations of security, model, asset, monitoring, and quality management systems.
The article outlines four key dimensions for evaluating data warehouse data quality—correctness, completeness, timeliness, and consistency—explains common consistency issues such as differing metric values across models, cross‑dimensional aggregations, and real‑time versus batch calculations, and proposes organizational and review mechanisms to mitigate these problems.
This article introduces the essential concepts of data quality management, covering the six key quality dimensions, detailed rule and metric templates, data profiling techniques, a systematic quality assurance workflow, and practical data cleaning methods to improve overall data governance.
This article shares Beike Zhaofang’s two‑year experience building a data governance center, covering the purpose and scope of governance, how the company tailors the focus to its business and system characteristics, the middle‑platform construction approach, project goal management, product and operation rollout, and the challenges and solutions encountered.
This article explains how ByteDance’s DataLeap platform tackles diverse data quality challenges across batch and streaming pipelines by defining quality dimensions, outlining a modular architecture, and sharing best‑practice optimizations for Spark, Flink and Presto‑based monitoring.
This article details Xiaomi's end‑to‑end metadata platform, covering its three‑layer architecture, the evolution of full‑domain metadata, real‑time lineage, precise measurement, and how these capabilities enable data map, governance, cost control, and quality improvements for future business empowerment.
The article explains common misconceptions about Master Data Management, emphasizing its enterprise-wide scope, the importance of data quality, governance, workflow, real‑time integration, and the need for organizational change management, while warning against treating MDM as a simple project.
This article outlines a comprehensive approach to monitoring and ensuring the accuracy and timeliness of real‑time data streams, detailing background challenges, solution design, implementation steps using Flink and automated testing, alert handling procedures, and future improvement plans.
This article explains the essential concepts of data quality management, covering data quality rules, key metrics, profiling techniques, governance mechanisms, and cleaning processes to help practitioners improve and sustain high‑quality data across its lifecycle.
The article details iQIYI’s data middle platform architecture and its comprehensive data governance practices, covering platform overview, data flow, unified standards, metadata management, production quality assurance, and future AI‑driven enhancements, illustrating how centralized data services improve reliability, efficiency, and security.
The article details iQIYI's comprehensive data governance initiative for event tracking (Pingback), covering definitions, timing, quality requirements, governance challenges, standardized specifications, coordinate management, testing and gray‑release processes, upgrade workflows, and data security measures that together reduced event volume by 40% and cut resource consumption in half.
This article details iQIYI's comprehensive data‑governance practice for event tracking, covering the definition of pingback, the need for governance, the governance framework, coordinate management, gray‑data handling, and the upgrade process that reduced tracking volume by 40% while cutting resource consumption in half.
At iTech Salon, iQIYI’s Peng Tao outlined a three‑layer data‑quality monitoring framework—pingback, middle, and business report layers—detailing anomaly‑detection techniques such as thresholds, statistical, correlation and Prophet forecasting, and announced future plans for intelligent rule generation and automated attribution to pinpoint root causes.
Airbnb’s Midas certification establishes a company‑wide, multi‑dimensional golden‑standard for data quality—covering accuracy, consistency, timeliness, cost, and completeness—by requiring collaborative design, automated health checks, and four review stages, ensuring certified data is reliable, well‑documented, and ready for reporting, experimentation, and machine‑learning.
This article explains what a data warehouse is, contrasts it with traditional databases, outlines how to design and build a warehouse—including model selection, subject‑area definition, bus matrix, layering, and data quality—while also covering related concepts such as data middle platforms, data lakes, metadata, and modeling techniques.
Effective data governance hinges on leadership commitment, clear policies, skilled teams, and integration into business processes, and this article outlines twenty actionable strategies—from securing executive support and embedding rules in systems to fostering data quality, visualization, and sustainable operations—to guide organizations toward successful governance.
The Hello algorithm testing team outlines its workflow—from data collection and cleaning through model training, evaluation, and deployment—while teaching machine‑learning fundamentals, detailing company‑wide use cases, defining key terms, and describing four testing capability dimensions covering data quality, service reliability, model performance, and system engineering.
Volcengine’s Data Quality Platform bridges the gap between data validation and resource‑intensive computation in large‑scale environments, offering unified stream‑batch monitoring, data exploration, comparison, and alerting across Hive, ClickHouse, Kafka, and more, while addressing scalability, latency, and resource optimization challenges.
Airbnb’s 2019 Data Quality Improvement Plan reorganized its data‑engineering workforce, introduced a dedicated data‑engineer role, adopted a decentralized Minerva‑based architecture with Spark pipelines, instituted rigorous testing, governance, and certification processes, and established SLAs and monitoring to ensure timely, trustworthy, well‑documented data across the enterprise.
This article details the background, challenges, and step‑by‑step reconstruction of Cloud Music's membership automation data warehouse, covering modeling standards, task decoupling, Spark performance tuning, and data quality checks to achieve higher efficiency and reliability.
DiDi’s Omega platform provides an end‑to‑end traffic‑data pipeline—from SDK collection through real‑time and offline ETL to storage and analysis—augmented by a detection service that measures loss, duplication and accuracy, achieving sub‑1% SDK loss, integrity tagging, comprehensive monitoring dashboards, and includes a senior data‑engineer hiring call.
This article presents a comprehensive case study of Meituan's hotel‑travel data governance, covering the background, challenges, strategic goals, standardized processes, technical systems, cost and security optimizations, measurable outcomes, and future plans for automated governance.
This article explains what data governance is, why it is needed, its objectives, core components, implementation methodology, required tools, and real‑world practices from Meituan Delivery and Ant Financial, illustrating how organized data management drives business value and risk control.
Youzan Data Report Center monitors offline batch and real‑time data pipelines using accuracy and timeliness rules, cross‑table checks, upstream‑downstream comparisons, and scheduled alerts to detect anomalies early; since 2021 it has generated over 25 alerts, and plans a unified data‑quality dashboard.
The article outlines Ctrip's data governance framework, detailing background challenges, metadata construction, cost and quality optimization techniques, data flow improvements, platform modules, health metrics, and concludes with a summary of achievements and future directions.
This article outlines Meituan's journey in data governance, detailing the challenges of data quality, cost, security, standardization and efficiency, and presenting a three‑stage roadmap—passive, proactive, and automated governance—along with concrete technical and organizational solutions.
This article provides an extensive overview of data governance in the big‑data era, covering common pitfalls, the role of metadata, data quality management, data standardization, and data asset management, and offers practical recommendations for organizations to implement effective governance practices.
The article explains the importance of data quality in the big data era, defines key quality dimensions such as completeness, uniqueness, validity, consistency, accuracy, timeliness, and credibility, and details how each dimension can be measured and enforced through specific constraints and validation rules.
Meituan’s hotel‑travel platform tackled exploding data‑quality, cost, efficiency, and security issues by establishing a full‑link governance framework—standardized processes, a Data Management Committee, and unified “One Model, One Logic, One Service, One Portal” systems—that cut per‑unit costs by ~40%, boosted engineer productivity over 60%, eliminated major security incidents, and set the stage for autonomous, AI‑driven data governance.
The talk by Beike senior engineer Yang Zongqiang details the evolution of the company's data development platform, covering background, three architecture upgrades, platform features such as metadata management, data integration, scheduling, quality assurance, and future directions for building an enterprise‑grade big‑data system.
The article describes how algorithm testing teams tackled data‑quality and interface‑semantic monitoring problems by building a unified business monitoring platform that checks table, storage and service consistency, validates response semantics, and, through dashboards, alerts and correction tools, quickly identified dozens of offline and online issues, guiding future reliability enhancements.
This article explains Youzan's data governance framework, covering the definition of data governance, the company's asset‑centric approach, quantitative quality scoring, cost‑based pricing formulas, billing and allocation mechanisms, continuous operational improvements, and the measurable outcomes achieved.
This article details NetEase Yanxuan's 2020 initiative to improve data task governance, describing identified pain points, the pre‑mid‑post framework for model, baseline, and incident handling, and the resulting products, processes, and future plans for a more reliable data warehouse.
This article examines the common pitfalls that cause Master Data Management (MDM) projects to fail, including underestimating effort, insufficient resources, overly ambitious scope, lack of data governance, excessive rules, and inadequate executive support, offering practical insights for successful implementation.
This article details how NetEase Yanxuan designed and implemented a comprehensive traffic system—including event‑tagging methods, a top‑down management framework, data‑quality controls, testing strategies, and attribution models—to turn fragmented user behavior into actionable e‑commerce insights.
The document proposes a data‑quality assurance framework for mobile‑app tracking points that automatically collects client‑side data, generates validation rules from historical samples, and runs automated tests on over 100 critical points—cutting manual verification from half a day to minutes and using tools such as Frida and AOP to detect missing or altered tracking data.
This article presents Sohu Intelligent Media's data warehouse construction practice, covering fundamental concepts, batch and real‑time processing, OLAP theory, multidimensional modeling, workflow management, data quality, metadata lineage, and security, with a focus on Apache Doris and a Lambda‑style architecture.
This article details NetEase Yanxuan's four‑stage journey—from establishing a business‑centric BI platform to ensuring data quality, empowering CXOs with mobile dashboards, and delivering scenario‑specific data products—highlighting the challenges faced, technical solutions implemented, and key takeaways for building enterprise data products.
Qualitis is a big‑data‑platform‑based data‑quality‑management service that defines, detects, and reports data‑set quality issues, featuring idempotent backend services, load‑balanced high‑availability, Zookeeper‑coordinated process synchronization, thread‑pool throttling, and clearly separated internal and external APIs.
The talk explores the current pain points of AI testing, outlines data‑quality analysis methods, highlights critical ETL and model‑testing considerations, and shares practical case studies and platform designs to improve machine‑learning quality assurance.
This article explores the concept of data quality, identifies ten common root causes, presents a comprehensive data quality management framework, outlines evaluation methods and key dimensions, and discusses future challenges and tools for improving data quality in large‑scale data environments.
This article outlines a comprehensive approach to building a financial data warehouse, covering background needs, modeling methodologies, a layered architecture (I, C, S, R), data quality monitoring, metadata management, and detailed naming and coding standards to ensure maintainable, high‑quality data pipelines.
The article describes how Dada Group created the Tianhe platform to centralize, standardize, and automate massive data‑tracking (埋点) requirements across multiple product lines, detailing its goals, architecture, core functions, current status, and future development directions.
This article outlines the end‑to‑end design of a financial data warehouse, covering background needs, modeling methodology choices, a layered architecture, data quality monitoring, metadata management, naming and coding standards, and future improvement directions.
This article presents a comprehensive design and implementation guide for a financial data warehouse, covering background needs, modeling methodology choices, a layered architecture, data quality monitoring, metadata management, naming and coding standards, and future development directions.
This article summarizes the objectives, design principles, application scenarios, layer‑by‑layer construction methods, quality assurance mechanisms, and supporting tools for building a real‑time data warehouse using Apache Flink, providing practical guidance for data engineers and architects.
Information governance is a program that ensures enterprise data accuracy, completeness, consistency, accessibility, and security by establishing business‑driven roles such as a data governance committee, data stewards, and data custodians, and by defining key responsibilities, processes, and metrics for data quality, privacy, and compliance.
This article explains the concepts of data modeling and architecture, including logical data, process, and rule modeling, various data model types, master data management principles, metadata categories, and data quality management practices, highlighting their roles in enterprise information systems.
This article outlines the importance of data quality in the big‑data era, defines evaluation criteria such as integrity, accuracy, consistency and timeliness, describes daily monitoring and reconciliation processes, and proposes technical solutions and challenges for building a comprehensive data‑quality monitoring platform.
This article details the evolution of 58 Tongcheng Bao's real‑time data warehouse, describing the initial Spark‑Streaming architecture, its limitations, and the redesign using Flink with a layered ODS‑DWD‑DWS‑APP model, data‑quality monitoring, join techniques, and the resulting improvements in latency and accuracy.
The article explains why continuous, proactive data maintenance is essential for maintaining data quality, supporting business outcomes, and preventing data decay, and outlines the key components, success factors, and steps to implement an effective ongoing data maintenance program.
The article explains how strong organizational structures and governance practices are essential for building a reliable, outcome‑driven enterprise data strategy, covering data scope, roles, standards, metrics, cultural change, and practical steps for successful implementation.
The article provides a comprehensive overview of banking data governance, covering core domains such as data models, metadata, standards, quality, lifecycle, distribution, exchange, security, and services, and explains how big‑data techniques can improve risk control, product innovation, and operational efficiency.
This article details the evolution of Ctrip's flight ticket data warehouse, describing its historical tech stack, current architecture—including Hive, Presto, ClickHouse, CrateDB, and Flink—data synchronization methods, layer design, quality monitoring, and a real‑time price‑monitoring use case.
Amid the pandemic‑driven surge in online video demand, Alibaba Youku built a comprehensive real‑time data quality assurance system—covering data content, consistency, correctness, availability, timeliness, performance testing, and automated intervention—to ensure that billions of video search results are delivered accurately and efficiently.
This article uses a vivid kite‑selling story to illustrate how user profiling, data tagging, and recommendation tactics can be combined to increase transaction volume, improve average order value, and avoid common pitfalls such as unclear goals, poor data quality, and unvalidated tags.
This article outlines Ctrip's flight ticket data warehouse evolution, current big‑data technology stack, data synchronization methods, layered architecture, quality monitoring system, and a real‑time price anomaly detection case, providing practical insights for building scalable, reliable data warehousing solutions.
This article systematically describes the challenges, design principles, modeling methods, layered architecture, implementation steps, and standards used in building a comprehensive user behavior data warehouse for 58.com, highlighting practical experiences and future improvement directions.
To address inaccuracies in traditional information exposure metrics, this article proposes adopting advertising visibility standards—defining visible exposure by pixel and time thresholds, implementing client-side logging, unique TID tracking, and ETL pipelines—to provide more reliable data for product strategy and user behavior analysis.
This article outlines common event‑logging quality problems, a systematic registration and real‑time validation framework built on Flink, configurable rule syntax, explainable results, continuous monitoring, targeted optimizations, and an evaluation model that together form a comprehensive quality‑center for big‑data platforms.
This article presents a comprehensive data quality model derived from ISO 9126, maps its characteristics to data testing, outlines practical testing methods and tool requirements, and demonstrates how to integrate quality checks into the data development lifecycle for reliable, efficient big‑data pipelines.
This article dispels five widespread AI misconceptions—from believing AI works like the human brain to thinking it is bias‑free—while offering practical guidance on recognizing AI limits, improving data quality, managing risks, and applying AI responsibly across industries.
