The article details how to add field‑lineage tracking to Spark SQL by creating a custom SparkSessionExtension that injects a check‑analysis rule and a parser, which capture INSERT statements, analyze the physical plan, and generate a JSON mapping of source‑to‑target fields for data governance.
This article details Alibaba's large‑scale data model governance initiative, analyzing current data issues, presenting a comprehensive solution—including model digitization, public model sinking, productization, daily governance, and search‑enhancement—and outlining achieved results and future plans to further improve data quality, reuse, and operational efficiency.
This article explains the challenges of early real‑time data pipelines, introduces a layered real‑time warehouse architecture, provides step‑by‑step Flink SQL code for building a demo warehouse, and covers comprehensive data governance, quality metrics, lifecycle management, and naming conventions for production‑grade big‑data systems.
This article explains ByteDance's data lineage architecture, covering data sources, processing pipelines, graph‑based modeling, key application scenarios, quality metrics such as accuracy, coverage and timeliness, and future directions for improving and standardizing lineage across its massive data platform.
DataWorks, Alibaba Cloud’s comprehensive PaaS platform, combined with the serverless MaxCompute data warehouse, offers an integrated solution for data integration, development, quality, and services, while detailed naming and layer conventions ensure scalable, maintainable big‑data architectures and effective governance across ODS, CDM, DWD, DWS, and ADS layers.
This article traces the evolution from traditional data warehouses to modern data lakes and data middle platforms, explains why siloed data development hampers efficiency, and details the architecture and indicator‑library design used by Zhengcaiyun to achieve unified, reusable data services.
This article outlines the comprehensive architecture and key points for building digital R&D management capabilities in financial organizations, reviewing historical challenges, identifying four major pain points, proposing an overall framework, detailing twelve essential capabilities, and offering principles for effective implementation.
This article presents MobTech’s comprehensive data governance and security practices, covering the necessity of governance, its benefits, a full‑chain governance framework, specific challenges in the financial sector, the evolution of their integrated architecture, and detailed implementations of security, model, asset, monitoring, and quality management systems.
This article presents MobTech's comprehensive data governance and security practices, covering the necessity of governance, challenges in large‑scale data environments, the full‑link governance chain, modular architecture, and specific implementations for financial risk‑control scenarios.
The article outlines four key dimensions for evaluating data warehouse data quality—correctness, completeness, timeliness, and consistency—explains common consistency issues such as differing metric values across models, cross‑dimensional aggregations, and real‑time versus batch calculations, and proposes organizational and review mechanisms to mitigate these problems.
The article explains why data maturity is essential for modern enterprises, defines its three pillars—people, tools, and readiness—shows how treating data as a product follows the same principles as great products, and outlines the four S (Speed, Scale, Simplicity, SQL) that guide a mature data ecosystem.
This comprehensive guide explains data‑warehouse construction standards, covering model architecture principles, layered design (ODS, DWD, DWM, DWS, APP), domain division, data‑model design rules, type and naming conventions, table lifecycle management, and practical implementation examples.
The article examines Amazon’s Intelligent Lakehouse architecture, tracing its evolution from early data‑lake‑warehouse integrations to a modern, serverless, secure, and AI‑enhanced platform that unifies data storage, governance, and analytics to lower big‑data costs and boost agility.
ByteDance’s data platform, built over seven years, now handles exabyte-scale data and over 100 million TPS, using a hybrid “middle‑platform + Business Partner” model, custom engines like ClickHouse/ByteHouse, agile governance, and a suite of products to support internal and external businesses, illustrating large-scale big-data engineering practices.
This comprehensive guide outlines a step‑by‑step learning roadmap for aspiring big data professionals, covering fundamentals, programming languages, Linux, databases, distributed theory, networking, offline and real‑time computing, data governance, warehouses, toolchains, video/book recommendations, future industry trends, interview tips, and community resources.
This article details Xiaomi's end‑to‑end metadata platform, covering its three‑layer architecture, the evolution of full‑domain metadata, real‑time lineage, precise measurement, and how these capabilities enable data map, governance, cost control, and quality improvements for future business empowerment.
This article details NetEase Cloud Music's comprehensive data governance journey, covering data warehouse architecture, data standards, event tracking (埋点) governance, asset lifecycle management, and future automation plans, illustrating how systematic governance improves data quality, cost efficiency, and business insight.
This article explains master data management concepts, regulatory drivers, business benefits, key technical challenges, architectural trends such as graph databases and machine learning, and highlights leading vendors, providing a comprehensive overview for enterprises seeking modern MDM solutions.
The article explains common misconceptions about Master Data Management, emphasizing its enterprise-wide scope, the importance of data quality, governance, workflow, real‑time integration, and the need for organizational change management, while warning against treating MDM as a simple project.
The article outlines master data management concepts and maturity levels, then details Baidu Smart Mini Program’s practical architecture—spanning analysis, domain‑driven design, high‑availability services, transaction handling, caching, real‑time sync, and governance—that eliminates data silos, ensures consistency, and supports over 9,000 QPS with 99.99% SLA.
This article outlines Ctrip Vacation's data standard management approach, covering why standards are needed, the three‑element framework of scope, tools, and policies, and detailed practices for data integration, production change handling, metadata governance, portal dashboard standardization, and self‑service query templating.
This article introduces the concept of a data map, explains its background and goals, describes Youzan’s end‑to‑end data‑map practice—including full data lineage, search, management, link analysis, impact estimation, and optimization—and concludes with a summary and future outlook.
This article details NetEase Yanxuan's real-time computing platform development from 2017 to present, covering its architecture, Flink‑SQL development environment, service‑oriented deployment, resource optimization, cloud‑native migration, comprehensive data governance, and future plans for stream‑batch integration and intelligent job diagnostics.
This article details NetEase Yanxuan's evolution of a real‑time data platform from 2017 to present, covering background, current scale, layered architecture, Flink‑SQL development IDE, service‑oriented task execution, resource‑optimizing deployment modes, cloud‑native migration, comprehensive data governance, and future batch‑stream integration plans.
This article outlines six practical strategies—identifying bottlenecks, empowering data engineers, adopting distributed management, creating data contracts, embracing diverse perspectives, and streamlining workflows—to help organizations leverage cloud data warehouses more efficiently and drive better business intelligence outcomes.
This article explains the essential concepts of data quality management, covering data quality rules, key metrics, profiling techniques, governance mechanisms, and cleaning processes to help practitioners improve and sustain high‑quality data across its lifecycle.
This article presents a practical re‑design of the DAMA‑DMBOK data management framework, integrating Huawei's data approach and over a decade of governance experience, and details twelve knowledge domains with resources, examples, and guidance for building a comprehensive data governance system.
This article compiles a comprehensive list of Chinese-language articles and tutorials covering big‑data technologies such as Flink, Spark, Hive, ClickHouse, data governance, and related interview preparation resources, providing a structured learning path for aspiring data engineers.
This article presents the OneData methodology for designing, standardizing, and governing a data warehouse, detailing background challenges, goals, industry references, core concepts, unified business and design consolidation, data modeling layers, naming conventions, data quality controls, and the resulting operational improvements and business value.
The article explains what a data lake is, its architecture and core capabilities, compares it with data warehouses, discusses its value and challenges, and reviews major open‑source platforms such as Delta Lake, Iceberg, and Hudi.
The article details iQIYI’s data middle platform architecture and its comprehensive data governance practices, covering platform overview, data flow, unified standards, metadata management, production quality assurance, and future AI‑driven enhancements, illustrating how centralized data services improve reliability, efficiency, and security.
The article examines how the Personal Information Protection Law creates a new authorization framework and introduces the “computable but not identifiable” concept, arguing that trusted‑computing technologies and controlled environments can reconcile strict privacy safeguards with the data‑driven needs of AI and other industries.
At Baidu Aifanfan, the data team built a unified real‑time and offline big‑data platform—leveraging Watt, Bigpipe, Fengge, AFS and Palo within Lambda/Kappa patterns and a fast‑slow parallel rollout—that cut OLAP query latency from 18 minutes to under 15 seconds, enabled self‑service analytics, and standardized metrics across 15 agile teams.
In a recent statement, ICBC’s CTO emphasizes that data, as a crucial production factor, derives its core value during use, urging dynamic data security and personal information protection, cross‑institution collaboration, regulated data markets, and safe cross‑border flows to support a healthy digital economy.
The presentation at the DataFun Summit detailed NetEase's data‑governance tool practice, reviewing past initiatives, current challenges, comprehensive product strategies, and future roadmap to improve compute and storage efficiency, cost quantification, and systematic governance across business lines.
This article explains how CITIC Bank's software development center built a data value system and management cockpit, detailing business objectives, overall architecture, digital management methodology, implementation steps, and real‑world usage to support the bank's digital transformation.
This article presents a comprehensive overview of CITIC Bank's data value system and cockpit construction, detailing business objectives, overall planning, digital management framework, methodology, implementation cases, and current usage, illustrating how data-driven analytics support the bank's digital transformation.
The article details iQIYI's comprehensive data governance initiative for event tracking (Pingback), covering definitions, timing, quality requirements, governance challenges, standardized specifications, coordinate management, testing and gray‑release processes, upgrade workflows, and data security measures that together reduced event volume by 40% and cut resource consumption in half.
This article details Baixin Bank’s multi‑dimensional analysis framework, covering the bank’s business model, data accuracy, completeness and usability requirements, the design of indicator and analysis systems, ladder‑style service concepts, user‑product‑enterprise scenario modeling, and the implementation of self‑service data products and governance processes.
This article details iQIYI's comprehensive data‑governance practice for event tracking, covering the definition of pingback, the need for governance, the governance framework, coordinate management, gray‑data handling, and the upgrade process that reduced tracking volume by 40% while cutting resource consumption in half.
At iTech Salon, iQIYI’s Peng Tao outlined a three‑layer data‑quality monitoring framework—pingback, middle, and business report layers—detailing anomaly‑detection techniques such as thresholds, statistical, correlation and Prophet forecasting, and announced future plans for intelligent rule generation and automated attribution to pinpoint root causes.
This article explains what a data warehouse is, contrasts it with traditional databases, outlines how to design and build a data warehouse—including model selection, topic domain division, bus matrix, layered architecture, and data governance—then expands to the concept of a data middle platform and its distinction from data lakes and big‑data platforms.
This week’s WecTeam Front‑end Weekly spotlights two technical deep‑dives: a Swift‑Objective‑C mixed‑compilation technique that slashes build times by 35%, and ByteDance’s large‑scale data‑tracking governance framework that underpins its trillion‑plus real‑time analytics pipeline.
This article explains the origins and purpose of data lakes, outlines four key construction goals, details common ingestion methods and storage technologies, and describes essential governance practices such as cataloging, data quality, and regulatory compliance.
This article presents Huawei's extensive digital transformation journey, detailing the continuous management system reforms, strategic shifts across multiple industries, data governance challenges, and practical initiatives such as cloud platforms, intelligent supply chains, and customer‑centric digital experiences that together illustrate how large enterprises can achieve sustainable growth through digitalization.
This article details Meituan Waimai's data warehouse evolution—covering business context, four‑layer architecture, Spark‑based ETL, successive V1.0, V2.0, and V3.0 redesigns, data governance practices, resource‑optimization tactics, security measures, and future road‑maps—illustrated with diagrams and concrete technical choices.
Effective data governance hinges on leadership commitment, clear policies, skilled teams, and integration into business processes, and this article outlines twenty actionable strategies—from securing executive support and embedding rules in systems to fostering data quality, visualization, and sustainable operations—to guide organizations toward successful governance.
The article explains how technology serves as the foundation and roadmap for digital transformation, while clear, value‑oriented rules and communication channels are essential to prevent failures and ensure cohesive, enterprise‑wide progress.
The speech highlights four major problems in China's consumer internet—zero‑sum competition, exploitative product design, unchecked data collection, and price discrimination—and argues that a shift toward industry‑specific internet models can create sustainable, multi‑party value.
This article details the historical development, architectural layers, ETL migration to Spark, data modeling standards, governance processes, resource optimization, security measures, and future roadmap of Meituan Waimai's offline data warehouse, illustrating how the team addressed scalability and efficiency challenges.
The article explains that digital transformation for traditional companies requires a mindset shift beyond mere analysis of status and environment, and offers a PPT detailing the 14th Five-Year Plan, transformation roadmap, data governance, and AI application to guide enterprises through this strategic overhaul.
This article, compiled from a Volcano Engine meetup, explains how ByteDance’s unified traffic platform designs, governs, and processes massive event‑tracking data in real time, covering embedding content solutions, link architecture, dynamic processing engines, and data‑governance practices that support trillions of daily events.
The article outlines Alibaba's twelve‑year journey building a data middle‑platform, detailing four development stages, the four major technical challenges faced, and emerging trends such as lake‑warehouse integration, autonomous data‑warehouse operation, natural‑language query, and AI‑driven data engineering.
The article outlines Alibaba's twelve‑year journey of building a data middle platform, describing four development stages, the technical challenges faced, and emerging trends such as lake‑warehouse integration, autonomous data‑warehouse operation, natural‑language query, and AI engineering.
This article explores how a clear enterprise data strategy—distinguishing strategic goals from tactical steps, emphasizing clean and governed data, and integrating analytics with business missions—drives reliable outcomes and supports the experience economy through coordinated CXM platforms and data products.
This article details NetEase Cloud Music's journey of redesigning its data warehouse and governance processes to support over a billion monthly active users, covering pain points, standardization, shared services, self‑service tools, and the resulting improvements in data quality, latency, and operational efficiency.
This comprehensive guide explains the concepts, architecture, and best practices of data middle platforms, covering data aggregation, ingestion tools, offline and real‑time development, data governance, service layers, and implementation details for building scalable big‑data solutions.
This article details JD Logistics’ challenges in handling petabyte‑level data, outlines their existing data architecture, and explains how they adopted Apache Doris for faster, scalable analytics, covering table management, data import workflows, visualization tools, and future roadmap for data engineering.
This article presents a comprehensive case study of Meituan's hotel‑travel data governance, covering the background, challenges, strategic goals, standardized processes, technical systems, cost and security optimizations, measurable outcomes, and future plans for automated governance.
This article explains what data governance is, why it is needed, its objectives, core components, implementation methodology, required tools, and real‑world practices from Meituan Delivery and Ant Financial, illustrating how organized data management drives business value and risk control.
The article details NetEase Yanxuan's end‑to‑end data product ecosystem and data middle platform, describing four core data products, the architecture of the data middle platform, efficient high‑quality delivery, governance practices, and key performance metrics that support data‑driven decision making.
ULTRON is a one‑stop real‑time data‑warehouse development platform built on FlinkSQL that unifies data integration, asset management, cluster deployment, modeling, ETL, OLAP analysis and governance, addressing the limitations of traditional batch‑oriented warehouses and simplifying streaming data workflows for developers.
This article explains what a big data platform is, outlines its seven‑component overall architecture, details the technical stack from data sources to applications, and describes the key subsystems such as catalog management, data integration, governance, storage, processing, sharing, development, and analysis.
The article outlines Ctrip's data governance framework, detailing background challenges, metadata construction, cost and quality optimization techniques, data flow improvements, platform modules, health metrics, and concludes with a summary of achievements and future directions.
This article explains how 58's commercial data team built a comprehensive data metric system—from identifying common metric definition issues to establishing a domain‑driven hierarchy, distinguishing atomic and derived metrics, implementing a unified metric management platform, and providing APIs and examples for querying and visualizing metrics.
This article provides a comprehensive overview of data warehouses, explaining their purpose, differences from databases, OLTP vs OLAP, traditional versus internet data warehouse models, layered architecture, modeling theories, metric dictionaries, date dimensions, naming conventions, data governance, and incremental synchronization techniques with practical SQL examples.
Huawei’s 2017 digital‑transformation vision led to a five‑step data‑governance blueprint that evolved through two phases, defining a detailed data‑classification framework, structured and unstructured data management methods, metadata governance, and compliance‑driven external data handling to support enterprise‑wide intelligent operations.
The 2021 GOPS Global Operations Conference in Shenzhen highlighted the shift toward intelligent, AI‑driven IT operations, presenting practical solutions, a three‑principle six‑step framework, and four core capabilities that help enterprises digitize, govern, and automate their operational data for higher efficiency.
This article outlines a comprehensive data model architecture, detailing layered data stores, classification structures, naming conventions, and core modeling principles to guide effective data warehouse design and implementation.
This article provides a comprehensive technical overview of data middle platforms, covering data aggregation, offline and real‑time development, smart operations, data asset management, governance, service layers, platform implementations, warehouse layering, and key differences between offline and real‑time data warehouses.
The article reviews Anthony J. Algmin’s reflections on past data‑architecture predictions, current hot topics such as cloud, AI/ML, data governance, and real‑time analytics, and forecasts future trends including metadata management, blockchain, and the evolving role of data architects within enterprises.
This article outlines Meituan's journey in data governance, detailing the challenges of data quality, cost, security, standardization and efficiency, and presenting a three‑stage roadmap—passive, proactive, and automated governance—along with concrete technical and organizational solutions.
This article explains the concept of a data lake—its origin in 2011, how it differs from traditional databases and data warehouses, its core characteristics such as raw data storage, on‑demand computing, and schema‑on‑read, as well as its advantages, challenges, architectural components, and future outlook within the big‑data ecosystem.
This article provides an extensive overview of data governance in the big‑data era, covering common pitfalls, the role of metadata, data quality management, data standardization, and data asset management, and offers practical recommendations for organizations to implement effective governance practices.
This article shares practical notes on building a one‑stop big data platform, outlining essential functions such as data extraction, cleaning, storage, analysis, governance, and security, and presents implementation case studies from WeBank, Beike, and iQIYI to illustrate real‑world architectures and solutions.
The article explains the importance of data quality in the big data era, defines key quality dimensions such as completeness, uniqueness, validity, consistency, accuracy, timeliness, and credibility, and details how each dimension can be measured and enforced through specific constraints and validation rules.
This article provides a detailed introduction to data middle platform concepts, covering data aggregation, ingestion tools, offline and real‑time development, data governance, service layers, monitoring, and deployment patterns, illustrating how enterprises build unified data ecosystems across various industries.
This article debunks eight common misconceptions about data lakes, explains why they are not mutually exclusive with data warehouses, clarifies that they are not limited to Hadoop or raw data only, and provides practical tips for building flexible, secure, and business‑driven data lake solutions.
Meituan’s hotel‑travel platform tackled exploding data‑quality, cost, efficiency, and security issues by establishing a full‑link governance framework—standardized processes, a Data Management Committee, and unified “One Model, One Logic, One Service, One Portal” systems—that cut per‑unit costs by ~40%, boosted engineer productivity over 60%, eliminated major security incidents, and set the stage for autonomous, AI‑driven data governance.
This article examines how result‑oriented DevOps drives digital transformation while exposing digital risks—from missing high‑level test scenarios and broken security data links to insufficient user‑experience foresight—and outlines strategies for data governance, risk mitigation, and effective decision‑support across the enterprise.
This article details a three‑part technical sharing that covers cost governance for offline Hadoop clusters, a large‑scale data‑center migration with architecture upgrades, and a tiered storage strategy using EC and COS to reduce storage costs and improve performance in a cloud‑native big‑data environment.
This article presents a comprehensive model quality assurance framework for click‑through‑rate (CTR) prediction, detailing the challenges of data and logic inconsistency, defining consistency goals, describing a full‑stack verification pipeline—including online data capture, offline sample alignment, multi‑stage q‑value comparison, and automated reporting—and sharing practical deployment experiences and results.
The article outlines Zhang Jindong’s recommendations for strengthening data security and governance in China, detailing legal background, Suning’s practical measures, and the need for clear data‑sharing attributes to enable safe, efficient public data sharing across industries.
The article examines China's push for free public data sharing, highlighting policy directives, the need for top‑level design, security standards, and education to create a unified, safe data‑governance framework that fuels the digital economy.
This article outlines Meituan's hotel‑travel data governance evolution, describing the key quality, cost, security, standardization and efficiency challenges faced as the business scaled, and detailing the organizational, technical, metric, service and product‑entry solutions implemented to achieve systematic, measurable, and automated data governance.
This article explains what data lineage is, why it is essential for data governance in large‑scale big‑data platforms, compares Apache Atlas with a custom solution, and details the technical choices, architecture, and performance optimizations behind the self‑built duo‑lineage system.
This article details NetEase Yanxuan's 2020 initiative to improve data task governance, describing identified pain points, the pre‑mid‑post framework for model, baseline, and incident handling, and the resulting products, processes, and future plans for a more reliable data warehouse.
NetEase Yanxuan tackled data‑task governance by establishing pre‑operation guarantees, baseline‑driven in‑operation controls, and post‑operation interventions, delivering stable task output, reduced alarms, lineage awareness, rapid incident recovery, and reusable best‑practice products that earned the 2020 Technology Sharing Co‑building Award.
This article provides a comprehensive overview of data lakes and data warehouses, explaining their definitions, architectures, differences, and practical use cases, while also covering related concepts such as OLTP/OLAP, ETL processes, data governance, and modern lakehouse solutions.
This article details Youzan's end‑to‑end big data security architecture, covering data lifecycle protection, classification, access control, auditing, backup, privacy safeguards, sensitive data detection, masking strategies, and compliance processes to ensure secure and compliant data handling across the platform.
The article explains master data management (MDM) as a framework for creating a single, reliable source of truth, outlines its growing business relevance, discusses key technical challenges such as data governance and scalability, and explores next‑generation architectures involving graph databases, big data, and machine learning.
The DTCC conference recap explores emerging data trends, multi‑model databases, governance frameworks, architecture migrations, NewSQL and MySQL high‑availability, distributed transaction challenges, AI‑driven operations, data middle‑platform debates, cloud‑native storage‑compute separation, and comprehensive data security across the full data lifecycle.
This article examines the common pitfalls that cause Master Data Management (MDM) projects to fail, including underestimating effort, insufficient resources, overly ambitious scope, lack of data governance, excessive rules, and inadequate executive support, offering practical insights for successful implementation.
The platform implements comprehensive metadata governance by extracting, standardizing, and ingesting basic, trend, resource, lineage, and task metadata from offline and real‑time systems via a Kafka‑based SDK, enabling unified storage, monitoring, alerts, and future automation to improve data asset visibility and quality.
This article provides a comprehensive overview of data middle platform concepts, covering data aggregation, collection tools, development modules, job scheduling, baseline control, heterogeneous storage, permission management, real‑time and offline processing, governance, services, and implementation details for building robust big‑data solutions.
This article explains how new‑retail companies can design and implement a data middle platform using Alibaba Cloud's DataWorks, covering business model analysis, technical architecture, layer‑by‑layer data modeling, governance, security, and the concrete benefits of turning raw data into actionable business insights.
This presentation explains how knowledge graphs serve as the foundation for new‑infrastructure initiatives, detailing the evolution of AI from perception to cognition, the role of big‑data centers, DIKW modeling, intelligent data governance, and the construction of a cognitive AI middle‑platform for industry applications.
This article summarizes Bao Yongjun’s presentation on JD.com’s end‑to‑end big data platform, covering its strategic value, industry trends, architectural design, development phases from scale‑out to intelligent real‑time processing, and future directions for a cloud‑native, AI‑driven data ecosystem.
The article summarizes Beike's one‑stop big data development platform, describing its data business background, the evolution from a simple Hadoop‑Kafka‑Hive stack to a metadata‑driven, asset‑oriented platform, and outlines current capabilities in data management, integration, scheduling, quality, openness, and future plans.
This article outlines the Yanxuan data warehouse’s layered architecture, the offline and real‑time development platforms, the comprehensive standards for metric definition, model design, and SQL development, and proposes a six‑dimensional evaluation system covering data norms, security, quality, stability, continuous improvement, and development efficiency.
