This article outlines Meituan's journey in data governance, detailing the challenges of data quality, cost, security, standardization and efficiency, and presenting a three‑stage roadmap—passive, proactive, and automated governance—along with concrete technical and organizational solutions.
This article provides an extensive overview of data governance in the big‑data era, covering common pitfalls, the role of metadata, data quality management, data standardization, and data asset management, and offers practical recommendations for organizations to implement effective governance practices.
The article explains the importance of data quality in the big data era, defines key quality dimensions such as completeness, uniqueness, validity, consistency, accuracy, timeliness, and credibility, and details how each dimension can be measured and enforced through specific constraints and validation rules.
Meituan’s hotel‑travel platform tackled exploding data‑quality, cost, efficiency, and security issues by establishing a full‑link governance framework—standardized processes, a Data Management Committee, and unified “One Model, One Logic, One Service, One Portal” systems—that cut per‑unit costs by ~40%, boosted engineer productivity over 60%, eliminated major security incidents, and set the stage for autonomous, AI‑driven data governance.
The talk by Beike senior engineer Yang Zongqiang details the evolution of the company's data development platform, covering background, three architecture upgrades, platform features such as metadata management, data integration, scheduling, quality assurance, and future directions for building an enterprise‑grade big‑data system.
The article describes how algorithm testing teams tackled data‑quality and interface‑semantic monitoring problems by building a unified business monitoring platform that checks table, storage and service consistency, validates response semantics, and, through dashboards, alerts and correction tools, quickly identified dozens of offline and online issues, guiding future reliability enhancements.
This article explains Youzan's data governance framework, covering the definition of data governance, the company's asset‑centric approach, quantitative quality scoring, cost‑based pricing formulas, billing and allocation mechanisms, continuous operational improvements, and the measurable outcomes achieved.
This article details NetEase Yanxuan's 2020 initiative to improve data task governance, describing identified pain points, the pre‑mid‑post framework for model, baseline, and incident handling, and the resulting products, processes, and future plans for a more reliable data warehouse.
This article examines the common pitfalls that cause Master Data Management (MDM) projects to fail, including underestimating effort, insufficient resources, overly ambitious scope, lack of data governance, excessive rules, and inadequate executive support, offering practical insights for successful implementation.
This article details how NetEase Yanxuan designed and implemented a comprehensive traffic system—including event‑tagging methods, a top‑down management framework, data‑quality controls, testing strategies, and attribution models—to turn fragmented user behavior into actionable e‑commerce insights.
The document proposes a data‑quality assurance framework for mobile‑app tracking points that automatically collects client‑side data, generates validation rules from historical samples, and runs automated tests on over 100 critical points—cutting manual verification from half a day to minutes and using tools such as Frida and AOP to detect missing or altered tracking data.
This article presents Sohu Intelligent Media's data warehouse construction practice, covering fundamental concepts, batch and real‑time processing, OLAP theory, multidimensional modeling, workflow management, data quality, metadata lineage, and security, with a focus on Apache Doris and a Lambda‑style architecture.
This article details NetEase Yanxuan's four‑stage journey—from establishing a business‑centric BI platform to ensuring data quality, empowering CXOs with mobile dashboards, and delivering scenario‑specific data products—highlighting the challenges faced, technical solutions implemented, and key takeaways for building enterprise data products.
Qualitis is a big‑data‑platform‑based data‑quality‑management service that defines, detects, and reports data‑set quality issues, featuring idempotent backend services, load‑balanced high‑availability, Zookeeper‑coordinated process synchronization, thread‑pool throttling, and clearly separated internal and external APIs.
The talk explores the current pain points of AI testing, outlines data‑quality analysis methods, highlights critical ETL and model‑testing considerations, and shares practical case studies and platform designs to improve machine‑learning quality assurance.
This article explores the concept of data quality, identifies ten common root causes, presents a comprehensive data quality management framework, outlines evaluation methods and key dimensions, and discusses future challenges and tools for improving data quality in large‑scale data environments.
This article outlines a comprehensive approach to building a financial data warehouse, covering background needs, modeling methodologies, a layered architecture (I, C, S, R), data quality monitoring, metadata management, and detailed naming and coding standards to ensure maintainable, high‑quality data pipelines.
The article describes how Dada Group created the Tianhe platform to centralize, standardize, and automate massive data‑tracking (埋点) requirements across multiple product lines, detailing its goals, architecture, core functions, current status, and future development directions.
This article outlines the end‑to‑end design of a financial data warehouse, covering background needs, modeling methodology choices, a layered architecture, data quality monitoring, metadata management, naming and coding standards, and future improvement directions.
This article presents a comprehensive design and implementation guide for a financial data warehouse, covering background needs, modeling methodology choices, a layered architecture, data quality monitoring, metadata management, naming and coding standards, and future development directions.
This article summarizes the objectives, design principles, application scenarios, layer‑by‑layer construction methods, quality assurance mechanisms, and supporting tools for building a real‑time data warehouse using Apache Flink, providing practical guidance for data engineers and architects.
Information governance is a program that ensures enterprise data accuracy, completeness, consistency, accessibility, and security by establishing business‑driven roles such as a data governance committee, data stewards, and data custodians, and by defining key responsibilities, processes, and metrics for data quality, privacy, and compliance.
This article explains the concepts of data modeling and architecture, including logical data, process, and rule modeling, various data model types, master data management principles, metadata categories, and data quality management practices, highlighting their roles in enterprise information systems.
This article outlines the importance of data quality in the big‑data era, defines evaluation criteria such as integrity, accuracy, consistency and timeliness, describes daily monitoring and reconciliation processes, and proposes technical solutions and challenges for building a comprehensive data‑quality monitoring platform.
This article details the evolution of 58 Tongcheng Bao's real‑time data warehouse, describing the initial Spark‑Streaming architecture, its limitations, and the redesign using Flink with a layered ODS‑DWD‑DWS‑APP model, data‑quality monitoring, join techniques, and the resulting improvements in latency and accuracy.
The article explains why continuous, proactive data maintenance is essential for maintaining data quality, supporting business outcomes, and preventing data decay, and outlines the key components, success factors, and steps to implement an effective ongoing data maintenance program.
The article explains how strong organizational structures and governance practices are essential for building a reliable, outcome‑driven enterprise data strategy, covering data scope, roles, standards, metrics, cultural change, and practical steps for successful implementation.
The article provides a comprehensive overview of banking data governance, covering core domains such as data models, metadata, standards, quality, lifecycle, distribution, exchange, security, and services, and explains how big‑data techniques can improve risk control, product innovation, and operational efficiency.
This article details the evolution of Ctrip's flight ticket data warehouse, describing its historical tech stack, current architecture—including Hive, Presto, ClickHouse, CrateDB, and Flink—data synchronization methods, layer design, quality monitoring, and a real‑time price‑monitoring use case.
Amid the pandemic‑driven surge in online video demand, Alibaba Youku built a comprehensive real‑time data quality assurance system—covering data content, consistency, correctness, availability, timeliness, performance testing, and automated intervention—to ensure that billions of video search results are delivered accurately and efficiently.
This article uses a vivid kite‑selling story to illustrate how user profiling, data tagging, and recommendation tactics can be combined to increase transaction volume, improve average order value, and avoid common pitfalls such as unclear goals, poor data quality, and unvalidated tags.
This article outlines Ctrip's flight ticket data warehouse evolution, current big‑data technology stack, data synchronization methods, layered architecture, quality monitoring system, and a real‑time price anomaly detection case, providing practical insights for building scalable, reliable data warehousing solutions.
This article systematically describes the challenges, design principles, modeling methods, layered architecture, implementation steps, and standards used in building a comprehensive user behavior data warehouse for 58.com, highlighting practical experiences and future improvement directions.
To address inaccuracies in traditional information exposure metrics, this article proposes adopting advertising visibility standards—defining visible exposure by pixel and time thresholds, implementing client-side logging, unique TID tracking, and ETL pipelines—to provide more reliable data for product strategy and user behavior analysis.
This article outlines common event‑logging quality problems, a systematic registration and real‑time validation framework built on Flink, configurable rule syntax, explainable results, continuous monitoring, targeted optimizations, and an evaluation model that together form a comprehensive quality‑center for big‑data platforms.
This article presents a comprehensive data quality model derived from ISO 9126, maps its characteristics to data testing, outlines practical testing methods and tool requirements, and demonstrates how to integrate quality checks into the data development lifecycle for reliable, efficient big‑data pipelines.
This article dispels five widespread AI misconceptions—from believing AI works like the human brain to thinking it is bias‑free—while offering practical guidance on recognizing AI limits, improving data quality, managing risks, and applying AI responsibly across industries.
The article explains how data serves as a critical asset for recommendation systems, outlining the necessary steps from understanding business problems and data dimensions to collection, cleaning, integration, and analysis, while distinguishing explicit and implicit user feedback and emphasizing data quality, timeliness, and relevance.
This article introduces the concept of anomaly detection, outlines common application scenarios such as ELT pipelines, feature engineering, A/B testing, and fraud detection, and reviews various detection methods—including statistical models, machine learning, rule‑based logic, and density‑based techniques—while discussing practical implementation considerations.
The article details Ant Financial’s comprehensive data quality governance framework, covering its architecture, challenges, implementation strategies, and real‑world case studies, illustrating how the company integrates data monitoring, AI‑driven self‑healing, and rigorous release controls to ensure high‑quality data across its platform.
This presentation details how Beike Rental leverages big data and machine‑learning techniques to detect non‑authentic listings, build a four‑criterion true‑listing model, develop pricing and image‑analysis models, and design a merchant credit scoring system that improves service quality and market efficiency.
This article explains how Ctrip's Ticket BU raised test coverage and engineering efficiency by enhancing test data quality, building information models, persisting data, creating a searchable data platform, and deploying backend traffic‑replay and frontend image‑comparison tools such as SnapDiff.
This article analyzes the common reasons CMDB initiatives collapse, shares Huawei's three‑phase journey from inception to value creation, and distills practical lessons on data consumption, accuracy, automation, visualization, and organizational execution for successful configuration management.
Accenture's 2018 Technology Vision outlines five pivotal trends—Citizen AI, Extended Reality, True Data, Large‑Scale Collaboration, and the Intelligent Internet—explaining how they reshape business models, customer relationships, and the underlying infrastructure needed for future growth.
This article details Meituan's DataMan platform, describing the background of data quality challenges, the eight-step PDCA-driven solution, architectural design, technical stack, monitoring standards, and the resulting improvements in data governance and operational efficiency across their massive data warehouse ecosystem.
Meituan’s DataMan platform provides a unified, closed‑loop data‑quality governance solution that collects demand, refines rules, executes monitoring across offline and real‑time jobs, tracks issues, and builds a knowledge base, improving completeness, accuracy, consistency, and timeliness while optimizing storage, reducing fault resolution time, and supporting data‑driven decisions.
Suning’s real‑time log analysis system integrates Flume, Kafka, Storm and Spark Streaming to collect, cleanse, and compute metrics like NDCG, ensuring low latency, high throughput, exact‑once processing, and robust data safety while supporting multi‑dimensional analytics on massive online‑offline traffic.
The article introduces data-ci, a SQL‑driven unit testing framework that lets engineers write, organize, and automate data validation tests for ETL pipelines, providing assertions, failure callbacks, coverage reporting, and CI integration to improve data quality and reliability.
This talk details Baidu Waimai's end‑to‑end ETL design, covering demand sources, data flow patterns, multi‑stage system evolution, storage choices, scheduling architecture, configuration‑driven processing, quality monitoring, and how data lineage enables transparent, self‑service data delivery.
At the 2nd China Big Data International Summit 2017, JD’s chief architect presented how machine‑learning techniques are applied across e‑commerce to improve product data quality, ensure compliance, resolve image‑text mismatches, automate category identification, restructure titles, and build a multi‑dimensional product knowledge graph.
This article breaks down the four essential elements of a model project—business, features, data, and model—explains their priority order, and provides practical guidance on aligning technical choices with core business goals to achieve efficient, high‑impact outcomes.
This article explains how Inceptor’s data auditing feature works, showing how to enable or disable table‑level error logging, interpret and query the Log Error Table, and manage its access permissions to ensure accurate query results despite dirty data.
This article describes how Baidu Waimai’s user‑profile offline data system was equipped with a highly automated, coroutine‑based testing framework that dramatically improves field‑value accuracy verification, test coverage, and execution efficiency across strategy, ES, and API layers.
This article outlines the importance of store data in O2O products and presents a comprehensive workflow—including single‑attribute rules, multi‑attribute cross‑validation, and auxiliary checks—to detect and remediate low‑quality or erroneous store information, thereby improving user experience.
The article outlines a comprehensive O2O data quality assurance workflow for online movie seat selection, detailing background challenges, a three‑stage process, evaluation metrics, and a concrete case study that demonstrates how real‑time data monitoring and issue handling improve user experience.
