The article details Meituan Waimai's offline data warehouse evolution from its initial V1.0 design through V2.0 improvements to the V3.0 modeling‑tool driven architecture, covering the four‑layer framework, Spark‑based ETL, data governance processes, resource optimization, security measures, and future development plans.
The article describes the design, architecture, and operational details of a unified logging framework at Ctrip's payment center, covering log production via a Log4j2 extension, Kafka‑Camus collection, Hive/ORC storage, MapReduce parsing optimizations, and governance strategies for massive daily TB‑scale data.
The article defines an indicator system as a structured set of interrelated metrics and dimensions, explains its lifecycle and hierarchy, presents OSM and AARRR models for construction, details metadata and dimension management, addresses business‑technical challenges, outlines a roadmap for implementation, and showcases DiDi’s deployment of thousands of indicators across dozens of domains.
This article explores the concept of data quality, identifies ten common root causes, presents a comprehensive data quality management framework, outlines evaluation methods and key dimensions, and discusses future challenges and tools for improving data quality in large‑scale data environments.
This article recounts SuNing’s step‑by‑step journey of designing and implementing a supply‑chain data middle platform, outlining its business rationale, DDD‑based domain modeling, layered system architecture, and practical deployment insights that illustrate how a tailored big‑data solution can enhance data services and governance.
This article provides an extensive summary of data middle platform concepts, covering data aggregation, collection tools, offline and real‑time development, data governance, service layers, warehouse construction, and operational practices, illustrating how enterprises build and manage a unified data ecosystem.
This article shares the practical experience and thinking behind Ctrip's vacation data governance project, covering team efficiency optimization, demand sorting, data domain definition, warehouse layering, unified dimension modeling, metric standardization, and the overall benefits of a centralized data governance framework.
The article surveys the Apache big‑data ecosystem, covering Hadoop’s storage and resource management, column stores HBase and Kudu, compute engines Spark, Flink, Impala, and Presto, coordination via ZooKeeper, ingestion with Sqoop and Flume, messaging Kafka, security Ranger and Sentry, metadata Atlas, OLAP Kylin, Hive, quality tool Griffin, notebooks Zeppelin, visualizations Superset and Tableau, the TPCx‑BB benchmark, and ends with an Alibaba analysis competition notice.
Information governance is a program that ensures enterprise data accuracy, completeness, consistency, accessibility, and security by establishing business‑driven roles such as a data governance committee, data stewards, and data custodians, and by defining key responsibilities, processes, and metrics for data quality, privacy, and compliance.
This article outlines the importance of data quality in the big‑data era, defines evaluation criteria such as integrity, accuracy, consistency and timeliness, describes daily monitoring and reconciliation processes, and proposes technical solutions and challenges for building a comprehensive data‑quality monitoring platform.
This comprehensive guide outlines the entire big data technology ecosystem, covering foundational techniques, data acquisition, transmission, storage, processing frameworks, governance, and the most popular tools and platforms that power modern large‑scale data solutions.
The article explains how a well‑defined technology roadmap, encompassing architecture, data governance, storage, analytics, and automation, is essential for aligning tools and techniques with business goals to achieve a successful, outcome‑driven enterprise data strategy.
The article explains how strong organizational structures and governance practices are essential for building a reliable, outcome‑driven enterprise data strategy, covering data scope, roles, standards, metrics, cultural change, and practical steps for successful implementation.
This article presents a comprehensive technical overview of the Real-time Data Platform (RTDP), detailing its overall architecture, component selection—including DBus, Kafka, Wormhole, Moonbox, and Davinci—design philosophies, functional features, and various deployment patterns such as synchronous, stream-processing, rotation, and intelligent modes.
The article examines how China’s push for public data sharing, led by Suning chairman Zhang Jindong, proposes a data governance committee, a unified sharing platform, security frameworks, and education initiatives to break information silos and accelerate the digital economy.
This talk reviews China's big‑data evolution from early enterprise data warehouses to modern data middle platforms, outlines core architectural components, technology selections, data development practices, lifecycle and quality management, and shares practical Q&A insights for building scalable, cost‑effective data infrastructures.
The article traces Youzan’s data warehouse from its chaotic early days lacking structure, through a 2016 Airflow‑driven construction phase that introduced layered ODS/DW/Data Mart architecture and naming standards, to a mature stage focused on efficiency, security, SparkSQL, dimensional modeling, metadata, and ongoing real‑time and governance challenges.
Meituan Delivery’s data‑governance framework combines a four‑layer warehouse architecture with comprehensive business, technical, security, and resource‑management standards, continuous metadata and security controls, and tools such as Wherehows and QuickSight, delivering standardized, secure, and easily shareable data while guiding future optimization and emerging‑technology adoption.
The article explains the concept, architecture, and key components of a data middle platform—including data aggregation, development, asset management, service systems, and operational and security mechanisms—while also promoting related books and a giveaway.
This article explains what a data middle platform is, outlines its construction process, discusses how to choose suitable products, and presents enterprise case studies, offering a comprehensive guide to building and leveraging a data middle platform for big‑data initiatives.
The article analyzes iQIYI’s big‑data middle‑platform strategy, detailing its origins, architecture, digital‑asset management, governance principles and how a unified, transparent, and compatible data platform enables user‑centric, scalable innovation across the entertainment ecosystem.
This article presents the OneData methodology for building a robust big‑data platform, detailing background challenges, goals, unified input and output strategies, model design, naming conventions, data‑cleaning rules, and the resulting business benefits and future outlook.
The article explains the concept of a data middle platform, why it matters for rapid demand response and resource integration, and outlines the distinct responsibilities and required skill sets of data product managers and data platform product managers within such ecosystems.
By adapting Alibaba’s OneData methodology, the project establishes a unified data‑warehouse architecture, standards, and governance framework—including consolidated business intake, standardized design layers, naming conventions, and delivery metrics—that resolves data‑quality issues, enhances scalability and reusability, and delivers faster, reliable data support for evolving business needs.
This article explains how to build a data warehouse from scratch, covering its definition, system and collaboration layers, ETL requirements, data layering design, modeling steps, common challenges, and governance practices such as temporary table management and coding standards.
While Hadoop remains a powerful but complex batch‑processing engine, Cloudera’s merger with Hortonworks and its pivot toward an enterprise data cloud—offering hybrid, multi‑cloud analytics, security, and governance—signals a strategic shift that keeps Hadoop relevant yet no longer central amid rising competitors like MongoDB and Elasticsearch.
This article presents a comprehensive case study of the Dada‑JD Daojia big data platform, detailing its evolution from a MySQL‑based warehouse to a multi‑layered One Data, One Platform, One Service, Many Apps architecture, the technical challenges faced, and the strategic approaches adopted to ensure coverage, accuracy, stability, and scalability.
The article explains what a data middle platform is, why it differs from data warehouses and data platforms, outlines its core capabilities, discusses the strategic and tactical considerations for building one, and examines the organizational, technical, and privacy challenges involved in its adoption.
This article details a bank’s practical experience designing and implementing a hierarchical CMDB, covering architecture, data standards, lifecycle management, accuracy controls, query visualization, performance tuning, and real‑world use cases for daily operations and private‑cloud management.
The article explains the emergence of data middle platforms in the era of digital and intelligent transformation, defines their architecture and functions, outlines four key reasons for building them—including data reuse, AI enablement, innovation acceleration, and talent development—and provides practical principles for successful implementation.
The article details Ant Financial’s comprehensive data quality governance framework, covering its architecture, challenges, implementation strategies, and real‑world case studies, illustrating how the company integrates data monitoring, AI‑driven self‑healing, and rigorous release controls to ensure high‑quality data across its platform.
This article presents a machine‑learning approach for detecting sensitive data fields in a data warehouse by combining a Wide & Deep network with a TextCNN architecture, detailing data exploration, model design, training strategies, performance results, and deployment workflow.
Facing growing data‑driven pressures, Didi designed a full‑link data tiered protection framework that defines classification standards, integrates data levels across the entire pipeline, and applies concrete safeguards and tooling to improve resource allocation, backup reliability, and overall data reliability.
This article explains the evolution of AIOps, outlines the data‑governance and integration challenges faced by large‑scale IT environments, and presents a step‑by‑step solution architecture—including algorithms, multi‑dimensional views, and role‑based workflows—to enable intelligent, automated operations.
The article examines three typical data collection problems faced by startups—unclear collection methods, chaotic tracking points, and poor collaboration between data and engineering teams—and offers practical strategies such as adopting full‑event models, appointing data architects, and securing top‑down support to achieve reliable, comprehensive analytics.
The article outlines eight key principles that enterprises should follow to harness big data responsibly, covering goal definition, strategic partnership, source identification, continuous communication, agile iteration, technology evaluation, cloud alignment, and talent development with security considerations.
The article presents a comprehensive overview of 360's Big Data Center evolution, covering business background, platform‑as‑a‑service architecture, data asset management, user‑profile unification, platform milestones, technical architecture, performance optimizations, online query capabilities, future plans, and a Q&A session.
Master data refers to high‑value core business entities such as customers, products, and accounts that are shared across multiple systems, and master data management (MDM) provides the standards, technologies, and processes to collect, integrate, cleanse, govern, and distribute this data consistently throughout an organization, improving data quality, compliance, and agility.
This article details Meituan's DataMan platform, describing the background of data quality challenges, the eight-step PDCA-driven solution, architectural design, technical stack, monitoring standards, and the resulting improvements in data governance and operational efficiency across their massive data warehouse ecosystem.
Meituan’s DataMan platform provides a unified, closed‑loop data‑quality governance solution that collects demand, refines rules, executes monitoring across offline and real‑time jobs, tracks issues, and builds a knowledge base, improving completeness, accuracy, consistency, and timeliness while optimizing storage, reducing fault resolution time, and supporting data‑driven decisions.
The article presents a comprehensive solution for reducing Hadoop cluster storage consumption by applying ZFS‑based transparent compression and data‑governance techniques, detailing the technical background, design choices, implementation steps, performance optimizations, and observed storage savings.
The article explains Alibaba's Data Security Maturity Model (DSMM), its partnership program, the involvement of 17 leading security firms, and how the model aims to improve data security capabilities across industries by establishing standardized assessment criteria and fostering ecosystem collaboration.
The article explains the strategic importance of the Chief Data Officer role, outlining how CDOs drive data‑driven innovation through a four‑stage data supply chain—data supply, logistics, science, and execution—to create competitive advantage and business growth.
Enterprise big‑data platforms face massive scale and complex metadata relationships, but using Transwarp Governor’s data lineage and impact analysis graphs enables precise tracing of data origins, rapid error localization, and prediction of downstream effects, dramatically improving data quality and governance efficiency.
This article examines how Baidu Waimai designed and evolved its big data platform, comparing traditional BI to modern 4V‑driven architectures, detailing database choices, OLTP/OLAP trade‑offs, data‑analysis team structures, and the essential steps for data governance and platformization.
The talk shares hands‑on experiences and best‑practice methods for traditional data‑warehouse processing, public and behavioral data handling in big‑data environments, and practical guidance for migrating legacy warehouses to modern Hadoop‑based platforms, emphasizing data governance, security, and performance optimization.
The article compiles detailed notes from the 2015 Internet+ Big Data Summit, highlighting how data monetization reshapes advertising, drives financial analytics, improves operational efficiency, and strengthens security, while presenting real‑world case studies, models, and practical recommendations from industry experts.
This article explains when to introduce database usage standards, how to implement them effectively, the benefits they bring, and the challenges of enforcement, illustrated with real‑world examples, code snippets, and a Q&A session with a senior DBA.
