This article outlines the background, development stages, architectural evolution, key features such as incremental updates and quality metrics, and future directions of the data lineage capability within Volcano Engine's DataLeap big‑data governance platform.
This technical presentation reviews the historical development of big data platforms, outlines the four generations of cloud data platform architectures, details the modern cloud‑native stack—including unified metadata, scheduling, and integration systems—and showcases a real‑world industrial manufacturing case with a Q&A session.
This article presents an in‑depth overview of data lineage at ByteDance, detailing the design of storage, display, abstraction, implementation, and storage layers, optimization techniques for real‑time updates and queries, open export methods, practical use cases across asset, development, governance, and security domains, and future directions.
This article explains NetEase DataFang's comprehensive view on data governance, detailing the problems it solves, the structure of a governance system, and concrete steps for implementing integrated data development and governance across enterprises.
This article examines the inherent complexity of building big‑data platforms, compares the emerging concepts of Data Fabric and Data Mesh, outlines their architectural features, technology stacks, and practical implementation challenges, and offers guidance on when each approach is appropriate.
This article presents a comprehensive overview of the intelligent metadata‑driven data governance framework implemented by Southern Power Grid Yunnan, detailing its background, challenges, architectural design, key AI‑enabled technologies, practical case studies, and the resulting business value for the power industry.
This article details Baidu's Cloud Storage (BOS) architecture, the challenges of its legacy metadata system, and the design of a new generation metadata engine that enables trillion‑object buckets, million‑QPS performance, hierarchical namespaces, and intelligent lifecycle management.
The article explains the importance of data governance for organizations handling big data, outlines narrow and broad governance approaches, presents strategic design principles, and shares practical case studies from leading companies, while also offering a downloadable ebook of governance strategies.
The article explains the importance of data governance for organizations handling big data, distinguishes narrow and broad governance approaches, outlines strategic principles, and presents case studies from companies like Tencent, SF Tech, Huolala, and NetEase to illustrate effective governance practices.
The article details ZTO Express's data governance journey, covering company background, drivers and goals, challenges such as data asset inventory, standardization, quality, and modeling, and outlines their multi‑layered governance framework, practical implementations in data quality, model and metadata, and future plans.
Data lineage in big‑data environments tracks how data moves and transforms—from source tables through SQL processing to final storage—enabling management tasks such as domain segmentation, performance tuning, anomaly detection, and dependency verification, with implementations ranging from simple regex extraction to robust AST parsing and optimization, as used by tools like Alibaba DataWorks and Apache Atlas.
This article explains how Wing Payment uses metadata as the foundation of its data‑governance practice, describing the challenges of data quality, efficiency, cost and security, the four‑step governance framework, the design of its metadata platform, and future directions such as multi‑source management and intelligent recommendation.
This article provides an in-depth overview of metadata concepts, their technical and business classifications, value in data management, applications such as data profiling and lineage, optimization techniques for compute and storage, lifecycle management, and comprehensive data quality assurance practices within large‑scale big data environments.
This article explains the concept of metadata, its role in data warehouses, why managing metadata is critical for building, maintaining, and scaling data warehouse systems, and outlines practical steps, use cases, and tools for effective metadata management.
Uber processes over 500,000 daily Presto queries across 20 clusters handling more than 50 PB of data, and by deploying Alluxio Local Cache on NVMe disks they raised cache‑hit rates from roughly 65% to over 90% while addressing real‑time partition updates, node churn, and cache‑size constraints.
The article presents an in‑depth overview of Tencent's Oula data governance platform, describing its construction goals, core capabilities, DataOps‑driven development workflow, unified metric store, data map services, and practical Q&A on asset health scoring and data lineage, illustrating a comprehensive end‑to‑end big‑data governance solution.
Tencent's Oula platform, launched in 2019, provides a DataOps‑driven, end‑to‑end data governance solution covering data discovery, asset factory, metric platform, and governance engine, and the talk details its construction goals, data development governance, unified metric system, data map, and Q&A on asset health and lineage.
Amid the explosive growth of big data, this article explains why systematic data governance—covering metadata, quality, lifecycle, and security—is essential for turning raw data into measurable business assets, reducing costs, and enhancing operational efficiency.
This article outlines Kuaishou's evolution of metadata management from its early Hive‑centric stage to a unified 2.0 platform, detailing system architecture, key technologies, challenges, and future 3.0 vision for low‑code, automated, and intelligent data governance.
The article examines why Hadoop's Distributed File System may become obsolete by detailing its three main shortcomings—deployment complexity, metadata memory limits, and high replication overhead—and explores how newer architectures and erasure coding could address these issues.
The article details Cloud Music’s data‑governance journey, covering early modeling standards, self‑service data tools, quality and metadata management, asset‑reuse improvements, and cost‑saving Spark optimizations, while sharing concrete metrics, processes, and the team’s systematic methodology.
The article details a NullPointerException that occurs when ShardingSphere starts, explains how null values in TiDB‑generated view metadata trigger the error, describes the step‑by‑step investigation and reproduction using TiUP, and discusses several possible fixes and a call for community contributions.
This article explores how structured domain model metadata, derived from domain‑driven design principles, can standardize business component descriptions, enable visual UML modeling, support low‑code and no‑code generation, and ultimately reduce development costs while accelerating delivery of enterprise support systems.
This article explores the design and engineering of Baidu’s TafDB, a distributed metadata database that powers cloud object and file storage, detailing its architecture, namespace evolution, transaction optimizations, garbage collection strategies, and clock mechanisms that enable trillion‑scale metadata and millions of QPS.
This article walks you through the fundamentals of data governance, explains metadata management concepts, compares traditional tools with DataHub, and provides a step‑by‑step tutorial for installing Docker, Python, and DataHub 0.8.20 on CentOS 7, ingesting MySQL metadata, and exploring the UI.
The article details NetEase Media's data governance framework, covering business overview, data‑warehouse architecture evolution, layered data services, metadata and asset management, cost‑control strategies, and a roadmap for automated, mature data governance driven by DAMA principles.
This article explores the evolution and architecture of cloud‑native data platforms, covering their historical roots, modern components such as storage layers, ingestion, processing, metadata, and consumption, and offers practical guidance on selecting tools, designing pipelines, and implementing best‑practice strategies for scalable, flexible data infrastructure.
This article details Meituan's accommodation data governance team's evolution toward an automated, systematic, and standardized governance framework, covering background challenges, the conceptualization of a comprehensive governance system, its practical implementation across standardization, digitization, and systematization, and the resulting operational benefits and future directions.
This article explains the concept of data governance, outlines ByteDance's four‑mission platform goals, details the end‑to‑end governance workflow, and describes the one‑stop, full‑link, and rule‑based architecture that powers their data governance solution.
This article explains the motivation, design principles, implementation details, and performance improvements of the new multi‑modal indexing subsystem introduced in Apache Hudi 0.11.0 for Lakehouse architectures, covering scalable metadata, ACID updates, fast lookups, file listing, data skipping, upsert performance, and future work.
This article explains the concept of a schema—from its Greek origins and psychological meaning to its role as metadata in databases and Kubernetes—detailing different database schema models, Kubernetes resource definitions, and how to extend and register custom schemas in Go.
The Apache Pulsar 2.10.0 release introduces automatic cluster failover, lazy‑loading producers, new TableView support, enhanced broker interceptors, enriched client authentication, Etcd metadata storage, and numerous bug fixes, offering developers and operators a more flexible and performant messaging platform.
The article details how Bilibili built a unified metadata infrastructure—including a URN‑based model, collection pipelines, quality assurance, storage in TiDB/ES/HugeGraph, and query services—to support data discovery, lineage, impact analysis, and governance across its growing data platform.
The article explains the emerging concept of Data Fabric (data weaving), its design principles, how it integrates metadata, knowledge graphs, and AI/ML to automate data integration across hybrid and multi‑cloud environments, and outlines four essential pillars that leaders must master to deliver business value.
Bilibili’s data platform consolidates scattered metadata into a unified URN‑based model stored across TiDB, Elasticsearch, and HugeGraph, offering batch‑pull and embedded collection, flexible SQL‑like queries, comprehensive lineage mapping, and powering data‑map, lineage‑map, and impact‑analysis tools while planning expanded quality assurance and self‑service dictionaries.
This article explains the concept of inodes in Unix/Linux filesystems, detailing their structure, stored metadata, size calculations, inode numbers, directory handling, hard and soft links, and the special behaviors that arise from separating file names from inode identifiers.
This article explains the concept of Aspect‑Oriented Programming (AOP) and demonstrates how Python decorators—both function‑based and class‑based—can be used to implement AOP features such as pre‑ and post‑execution logic, handling arguments, preserving metadata with functools.wraps, and stacking multiple decorators.
This article describes Meituan's delivery data governance journey, explaining how systematic, metadata‑driven modeling unifies data definition, model design, and data production to break data silos, improve data quality, and enable self‑service analytics across the organization.
This article explains ByteDance's data lineage architecture, covering data sources, processing pipelines, graph‑based modeling, key application scenarios, quality metrics such as accuracy, coverage and timeliness, and future directions for improving and standardizing lineage across its massive data platform.
This article explains what Java annotations are, how they serve as metadata for classes, methods, and fields, outlines their common applications such as documentation generation and compile‑time checks, and details built‑in, meta‑ and custom annotations with code examples.
This article explains data governance fundamentals, why it is needed, its objectives, core components, implementation methodology, required tools, and real‑world practices from Meituan and Ant Financial, providing a comprehensive guide for managing data as a strategic enterprise asset.
This article details Xiaomi's end‑to‑end metadata platform, covering its three‑layer architecture, the evolution of full‑domain metadata, real‑time lineage, precise measurement, and how these capabilities enable data map, governance, cost control, and quality improvements for future business empowerment.
This article explains the methodology behind tag systems, defines their purpose and structure, introduces three key dimensions for constructing a tag framework, outlines the tag lifecycle and attributes, and provides a practical design template for implementation.
This article describes the evolution from an unsecured data warehouse to a comprehensive big‑data access control system at BanYu, detailing the background, data access methods, design goals, authentication and authorization mechanisms, policy configuration, integration with Metabase, and the overall workflow that balances security with efficiency.
This article introduces the cloud‑native LakeHouse solution from China Mobile Cloud, covering its lake‑warehouse integration concept, overall architecture, core functions such as storage‑compute separation, one‑click data ingestion, intelligent metadata discovery, serverless execution, JDBC support, incremental updates, and typical application scenarios in public and private clouds.
The article details iQIYI’s data middle platform architecture and its comprehensive data governance practices, covering platform overview, data flow, unified standards, metadata management, production quality assurance, and future AI‑driven enhancements, illustrating how centralized data services improve reliability, efficiency, and security.
This article explains data governance—from its definition and why it’s needed, to its goals, core components, PDCA‑based methodology, essential tools, and real‑world implementations at Meituan and Ant Financial—providing a comprehensive guide for organizations seeking to manage and leverage their data assets effectively.
This article explains the differences between the Dble commands reload @@config_all and reload @@metadata, analyzes why metadata errors occur after table changes without configuration updates, and provides step‑by‑step guidance to synchronize metadata correctly using these commands and check full @@metadata.
This article walks you through the origins, types, and practical uses of TypeScript decorators, explains metadata reflection, demonstrates decorator factories, and shows how they enable inversion of control and dependency injection in modern backend frameworks like NestJS and Midway.
This article explains what a data warehouse is, contrasts it with traditional databases, outlines how to design and build a warehouse—including model selection, subject‑area definition, bus matrix, layering, and data quality—while also covering related concepts such as data middle platforms, data lakes, metadata, and modeling techniques.
This article describes how Ctrip built a data lineage system for its big data platform, covering the concept of data lineage, collection methods, open‑source tools such as Apache Atlas and DataHub, the in‑house table‑level and field‑level solutions, implementation details for Hive, Spark and Presto, storage in JanusGraph, and practical applications in data governance, metadata management, scheduling and sensitivity labeling.
The article explains the concept of a data middle platform, describing its architecture, the essential big‑data foundation, metadata management, data service components such as BI and tag systems, and how these layers together enable unified data access, governance, and business intelligence across enterprises.
This article explains how object storage emerged as a cloud-native solution that surpasses traditional DAS, SAN, and NAS architectures by offering virtually unlimited capacity, robust metadata handling, and simple RESTful APIs for modern applications and large‑scale data workloads.
MySQL 8.0 replaces duplicated server‑level and engine‑level metadata with a unified data dictionary stored in InnoDB, introduces a two‑level cache (local and shared) built on templated hash maps, and provides atomic DDL operations, dramatically improving metadata consistency, performance, and management simplicity.
This article explains the logical layered architecture of modern data warehouses, covering data sources, ODS, DW/DWS layers, collection, storage on HDFS, synchronization tools, dimensional modeling (star, snowflake, constellation), metadata management, and task scheduling and monitoring, highlighting best practices for scalable big‑data solutions.
This article provides a comprehensive overview of distributed file systems, covering their historical evolution, essential requirements such as POSIX compliance, persistence, scalability, and security, and comparing centralized (e.g., GFS) and decentralized (e.g., Ceph) architectures along with strategies for high availability, performance optimization, and data consistency.
This article explains what data governance is, why it is needed, its objectives, core components, implementation methodology, required tools, and real‑world practices from Meituan Delivery and Ant Financial, illustrating how organized data management drives business value and risk control.
This article introduces Apache Kafka’s KRaft (Kafka Raft) mode, explains its architectural differences from ZooKeeper‑based deployments, details essential configuration parameters, and provides a complete step‑by‑step procedure—including commands and utility tools—to set up and operate a KRaft cluster.
The article outlines Ctrip's data governance framework, detailing background challenges, metadata construction, cost and quality optimization techniques, data flow improvements, platform modules, health metrics, and concludes with a summary of achievements and future directions.
Huawei’s 2017 digital‑transformation vision led to a five‑step data‑governance blueprint that evolved through two phases, defining a detailed data‑classification framework, structured and unstructured data management methods, metadata governance, and compliance‑driven external data handling to support enterprise‑wide intelligent operations.
This article explains a novel Swift hooking technique that modifies the virtual function table (VTable) to replace method implementations, detailing Swift's runtime structures such as TypeContext, Metadata, OverrideTable, and providing concrete ARM64 assembly and Swift code examples.
This article provides an extensive overview of data governance in the big‑data era, covering common pitfalls, the role of metadata, data quality management, data standardization, and data asset management, and offers practical recommendations for organizations to implement effective governance practices.
The article describes a backend‑for‑frontend architecture that pushes GraphQL into the BFF layer, separates fetch and display units, drives execution with metadata, unifies query models, applies caching and parallel processing optimizations, and demonstrates over 50 % logic reuse and doubled development efficiency in production.
This article presents JuiceFS, an open‑source cloud‑native distributed file system that addresses the limitations of object storage for big‑data and AI workloads by providing strong consistency, high‑performance metadata, multi‑protocol support, small‑file management, and deep Kubernetes integration.
Apache Hudi is being re‑positioned from a simple table format to a full‑featured data lake platform, offering transactional storage, MVCC concurrency, metadata services, Deltastreamer ingestion, and plans for cache and timeline metadata services, aligning its vision with modern lakehouse architectures.
The talk by Beike senior engineer Yang Zongqiang details the evolution of the company's data development platform, covering background, three architecture upgrades, platform features such as metadata management, data integration, scheduling, quality assurance, and future directions for building an enterprise‑grade big‑data system.
This article describes how NetEase developed a Hive metadata migration and merging tool that consolidates metadata from multiple independent Hive clusters into a single Hive metastore without moving HDFS data, detailing the challenges, ID handling, database operations, and step‑by‑step migration process.
This article introduces Apache Iceberg as an open table format for massive analytic datasets, explains its underlying concepts such as schema, partitioning, statistics, and read/write APIs, compares it with Hive Metastore, outlines its ACID commit process, highlights the performance and operational advantages for big‑data workloads, and previews upcoming community features.
This article explains the concepts and practices of data modeling, data architecture, master data management, metadata management, and data quality management, describing their roles, key components, and how they support reliable and consistent enterprise information systems.
This article explains how a metadata‑driven multi‑tenant data model decouples logical and physical schemas, enabling rapid SaaS product rollout, seamless scaling, fine‑grained customization, and zero‑downtime schema changes while ensuring data isolation, security, and high performance across millions of tenants.
This article explains the fundamentals and value of metadata, describes a comprehensive metadata management system and its layered architecture, outlines key technologies such as automatic SQL metadata extraction, and showcases practical applications like metadata query, impact analysis, data lineage, and business‑driven data needs within modern data warehouses.
This article presents Hulu's multi‑layered approach to content understanding and embedding, describing tag‑based graph embeddings, metadata‑BERT enhancements, multimodal video/audio feature aggregation, and various applications such as similarity search, ranking, cold‑start retrieval, and collection modeling, while also discussing current limitations and open research questions.
The article provides a comprehensive overview of the Qirin big‑data platform, detailing its architecture, core modules such as resource management, metadata, data ingestion, task development, interactive query, and self‑service analysis, and outlines future development plans for the system.
This article introduces the development history, current architecture, abstraction methods, and key application scenarios of Kuaishou's metadata platform, highlighting challenges such as heterogeneous data integration, large-scale asset management, and the platform's role in data search, lineage, governance, and future enhancements.
The article introduces the Qirin big data platform—a one‑stop solution covering resource management, metadata, data ingestion, task development, interactive querying, and self‑service analysis—detailing its modular architecture, typical processing workflow, and future development plans for enterprise‑wide data services.
The platform implements comprehensive metadata governance by extracting, standardizing, and ingesting basic, trend, resource, lineage, and task metadata from offline and real‑time systems via a Kafka‑based SDK, enabling unified storage, monitoring, alerts, and future automation to improve data asset visibility and quality.
This article details iQIYI's transition from a fragmented Data Warehouse 1.0 to a unified, standardized Data Warehouse 2.0, covering layered architecture, dimension and metric design, modeling workflows, metadata management, data lineage, and upcoming intelligent and automated data platform initiatives.
This article presents a comprehensive overview of Hive SQL lineage parsing, detailing the challenges of data provenance in large‑scale data warehouses, introducing ANTLR‑based parsing techniques, and describing a series of optimizations—including AST pruning, CTE handling, UDF registration, and metadata service integration—to improve both table‑level and column‑level lineage extraction and visualization.
File systems risk data corruption during power loss or crashes because writes are not atomic, so journaling—recording intended operations in a write‑ahead log before committing them—ensures metadata and user data consistency, with variations like data journaling and ordered (metadata) journaling improving performance and reliability.
The article explains how Alibaba Cloud's Data Lake Building Service enables fine‑grained lake management by integrating multiple compute engines—including EMR, MaxCompute, Blink, Hologres, PAI, and open‑source Hive, Spark, and Presto—through unified metadata and OSS storage, while outlining current features, special format support, and planned future enhancements.
The talk chronicles iQIYI’s shift from a fragmented five‑layer Data Warehouse 1.0 to a unified 2.0 architecture featuring a central Dimension Layer, business‑focused data marts, and subject‑oriented warehouses, while detailing platform services, rigorous metadata management, lineage tracking, and future goals of intelligent, automated, service‑oriented, model‑driven data governance.
Journal file systems use write‑ahead logging to record each write operation as a transaction, ensuring that after power loss or crashes the system can replay logs and maintain metadata and user‑data consistency, avoiding corruption and space waste through techniques like data, ordered, and metadata journaling.
This article explains how Kafka relies on ZooKeeper to store cluster metadata, detailing the ZK node hierarchy, the process by which clients locate brokers, the broker‑side handling of metadata requests, and recommended practices for large‑scale deployments.
The Metadata Center, built by Alibaba Entertainment, streamlines development by offering a searchable Data Source Plaza and a configurable Custom Interface engine that abstracts service calls, adds unified monitoring and circuit‑breaker safeguards, and leverages optimized scripting, cutting upfront coding effort and accelerating feature delivery across Youku applications.
This article explains the fundamentals of object storage, compares it with block and file storage, outlines its architecture, components, advantages, use cases, and limitations, showing why it has become the dominant storage model in modern cloud environments.
This article presents a comprehensive guide to constructing real-time data warehouses using Apache Flink, covering the motivations, design principles, application scenarios, layer-by-layer architecture, metadata and lineage management, quality assurance, and the supporting toolchain for reliable streaming analytics.
This article describes the background, metadata concepts, SPU and category‑attribute modeling, MySQL tree‑storage techniques, dynamic business configuration, rule‑engine and workflow‑engine solutions adopted in Tripyun‑Ctrip Cloud to build a reusable, scalable middle‑platform for the tourism industry.
This article explains why metadata management is crucial for storage systems and reviews four typical approaches—initial external‑DB storage, in‑memory loading, partitioned services with a proxy layer, and tiered caching/persistence—illustrated with diagrams and real‑world examples.
The article explains Alluxio's tiered metadata management architecture, describing how the system separates hot and cold metadata into cached and persisted layers, and details the custom asynchronous eviction thread and cache implementation that replace Guava cache for efficient large‑scale metadata handling.
This article explains the concepts of data modeling and architecture, including logical data, process, and rule modeling, various data model types, master data management principles, metadata categories, and data quality management practices, highlighting their roles in enterprise information systems.
The article provides a comprehensive overview of banking data governance, covering core domains such as data models, metadata, standards, quality, lifecycle, distribution, exchange, security, and services, and explains how big‑data techniques can improve risk control, product innovation, and operational efficiency.
The article traces Youzan’s data warehouse from its chaotic early days lacking structure, through a 2016 Airflow‑driven construction phase that introduced layered ODS/DW/Data Mart architecture and naming standards, to a mature stage focused on efficiency, security, SparkSQL, dimensional modeling, metadata, and ongoing real‑time and governance challenges.
Meituan Delivery’s data‑governance framework combines a four‑layer warehouse architecture with comprehensive business, technical, security, and resource‑management standards, continuous metadata and security controls, and tools such as Wherehows and QuickSight, delivering standardized, secure, and easily shareable data while guiding future optimization and emerging‑technology adoption.
This article explains Unix/Linux inodes—the metadata structures that store file information—covering their purpose, contents, size considerations, inode numbers, directory handling, hard and soft links, and special inode-related operations, with practical command examples and visual illustrations.
The article surveys modern big‑data architecture, contrasting Lambda and Kappa models, highlights common governance and integration pain points, and proposes an end‑to‑end platform featuring unified metadata, stream‑batch processing, one‑click ingestion, standardized modeling, intelligent query abstraction, and a comprehensive development IDE.
This tutorial explains how to use Spring Boot's Configuration Processor to generate JSON metadata for configuration properties, covering dependency setup, Java bean definitions, property files, tests, and how the resulting metadata improves IDE auto‑completion and documentation.
This article provides a comprehensive guide to Kubernetes resource metadata, covering labels, annotations, ownerReferences, selector syntax, practical kubectl operations, and the underlying controller pattern that drives declarative API behavior in cloud‑native environments.
This tutorial explains Kubernetes resource metadata—including spec, status, labels, annotations, and ownerReferences—demonstrates how to query and modify them with kubectl, and dives into the controller pattern, control loops, and the advantages of declarative APIs over imperative ones.
The article explains the MFS distributed file system, detailing its four components—Master, Metalogger, Chunkserver, and Client—along with hardware recommendations, metadata handling, replication strategies, and FUSE‑based client mounting, providing a comprehensive guide to building a GoogleFS‑like storage cluster.
This article explains how to design a high‑performance feed‑stream architecture—including product definition, data modeling, storage choices, synchronization modes, metadata handling, commenting, likes, sorting, search, and deletion—so that a system can support tens of millions to billions of users while remaining reliable and scalable.
