The article explains the emerging concept of Data Fabric (data weaving), its design principles, how it integrates metadata, knowledge graphs, and AI/ML to automate data integration across hybrid and multi‑cloud environments, and outlines four essential pillars that leaders must master to deliver business value.
Bilibili’s data platform consolidates scattered metadata into a unified URN‑based model stored across TiDB, Elasticsearch, and HugeGraph, offering batch‑pull and embedded collection, flexible SQL‑like queries, comprehensive lineage mapping, and powering data‑map, lineage‑map, and impact‑analysis tools while planning expanded quality assurance and self‑service dictionaries.
This article explains the concept of inodes in Unix/Linux filesystems, detailing their structure, stored metadata, size calculations, inode numbers, directory handling, hard and soft links, and the special behaviors that arise from separating file names from inode identifiers.
This article explains the concept of Aspect‑Oriented Programming (AOP) and demonstrates how Python decorators—both function‑based and class‑based—can be used to implement AOP features such as pre‑ and post‑execution logic, handling arguments, preserving metadata with functools.wraps, and stacking multiple decorators.
This article describes Meituan's delivery data governance journey, explaining how systematic, metadata‑driven modeling unifies data definition, model design, and data production to break data silos, improve data quality, and enable self‑service analytics across the organization.
This article explains ByteDance's data lineage architecture, covering data sources, processing pipelines, graph‑based modeling, key application scenarios, quality metrics such as accuracy, coverage and timeliness, and future directions for improving and standardizing lineage across its massive data platform.
This article explains what Java annotations are, how they serve as metadata for classes, methods, and fields, outlines their common applications such as documentation generation and compile‑time checks, and details built‑in, meta‑ and custom annotations with code examples.
This article explains data governance fundamentals, why it is needed, its objectives, core components, implementation methodology, required tools, and real‑world practices from Meituan and Ant Financial, providing a comprehensive guide for managing data as a strategic enterprise asset.
This article details Xiaomi's end‑to‑end metadata platform, covering its three‑layer architecture, the evolution of full‑domain metadata, real‑time lineage, precise measurement, and how these capabilities enable data map, governance, cost control, and quality improvements for future business empowerment.
This article explains the methodology behind tag systems, defines their purpose and structure, introduces three key dimensions for constructing a tag framework, outlines the tag lifecycle and attributes, and provides a practical design template for implementation.
This article describes the evolution from an unsecured data warehouse to a comprehensive big‑data access control system at BanYu, detailing the background, data access methods, design goals, authentication and authorization mechanisms, policy configuration, integration with Metabase, and the overall workflow that balances security with efficiency.
This article introduces the cloud‑native LakeHouse solution from China Mobile Cloud, covering its lake‑warehouse integration concept, overall architecture, core functions such as storage‑compute separation, one‑click data ingestion, intelligent metadata discovery, serverless execution, JDBC support, incremental updates, and typical application scenarios in public and private clouds.
The article details iQIYI’s data middle platform architecture and its comprehensive data governance practices, covering platform overview, data flow, unified standards, metadata management, production quality assurance, and future AI‑driven enhancements, illustrating how centralized data services improve reliability, efficiency, and security.
This article explains data governance—from its definition and why it’s needed, to its goals, core components, PDCA‑based methodology, essential tools, and real‑world implementations at Meituan and Ant Financial—providing a comprehensive guide for organizations seeking to manage and leverage their data assets effectively.
This article explains the differences between the Dble commands reload @@config_all and reload @@metadata, analyzes why metadata errors occur after table changes without configuration updates, and provides step‑by‑step guidance to synchronize metadata correctly using these commands and check full @@metadata.
This article walks you through the origins, types, and practical uses of TypeScript decorators, explains metadata reflection, demonstrates decorator factories, and shows how they enable inversion of control and dependency injection in modern backend frameworks like NestJS and Midway.
This article explains what a data warehouse is, contrasts it with traditional databases, outlines how to design and build a warehouse—including model selection, subject‑area definition, bus matrix, layering, and data quality—while also covering related concepts such as data middle platforms, data lakes, metadata, and modeling techniques.
This article describes how Ctrip built a data lineage system for its big data platform, covering the concept of data lineage, collection methods, open‑source tools such as Apache Atlas and DataHub, the in‑house table‑level and field‑level solutions, implementation details for Hive, Spark and Presto, storage in JanusGraph, and practical applications in data governance, metadata management, scheduling and sensitivity labeling.
The article explains the concept of a data middle platform, describing its architecture, the essential big‑data foundation, metadata management, data service components such as BI and tag systems, and how these layers together enable unified data access, governance, and business intelligence across enterprises.
This article explains how object storage emerged as a cloud-native solution that surpasses traditional DAS, SAN, and NAS architectures by offering virtually unlimited capacity, robust metadata handling, and simple RESTful APIs for modern applications and large‑scale data workloads.
MySQL 8.0 replaces duplicated server‑level and engine‑level metadata with a unified data dictionary stored in InnoDB, introduces a two‑level cache (local and shared) built on templated hash maps, and provides atomic DDL operations, dramatically improving metadata consistency, performance, and management simplicity.
This article explains the logical layered architecture of modern data warehouses, covering data sources, ODS, DW/DWS layers, collection, storage on HDFS, synchronization tools, dimensional modeling (star, snowflake, constellation), metadata management, and task scheduling and monitoring, highlighting best practices for scalable big‑data solutions.
This article provides a comprehensive overview of distributed file systems, covering their historical evolution, essential requirements such as POSIX compliance, persistence, scalability, and security, and comparing centralized (e.g., GFS) and decentralized (e.g., Ceph) architectures along with strategies for high availability, performance optimization, and data consistency.
This article explains what data governance is, why it is needed, its objectives, core components, implementation methodology, required tools, and real‑world practices from Meituan Delivery and Ant Financial, illustrating how organized data management drives business value and risk control.
This article introduces Apache Kafka’s KRaft (Kafka Raft) mode, explains its architectural differences from ZooKeeper‑based deployments, details essential configuration parameters, and provides a complete step‑by‑step procedure—including commands and utility tools—to set up and operate a KRaft cluster.
The article outlines Ctrip's data governance framework, detailing background challenges, metadata construction, cost and quality optimization techniques, data flow improvements, platform modules, health metrics, and concludes with a summary of achievements and future directions.
Huawei’s 2017 digital‑transformation vision led to a five‑step data‑governance blueprint that evolved through two phases, defining a detailed data‑classification framework, structured and unstructured data management methods, metadata governance, and compliance‑driven external data handling to support enterprise‑wide intelligent operations.
This article explains a novel Swift hooking technique that modifies the virtual function table (VTable) to replace method implementations, detailing Swift's runtime structures such as TypeContext, Metadata, OverrideTable, and providing concrete ARM64 assembly and Swift code examples.
This article provides an extensive overview of data governance in the big‑data era, covering common pitfalls, the role of metadata, data quality management, data standardization, and data asset management, and offers practical recommendations for organizations to implement effective governance practices.
The article describes a backend‑for‑frontend architecture that pushes GraphQL into the BFF layer, separates fetch and display units, drives execution with metadata, unifies query models, applies caching and parallel processing optimizations, and demonstrates over 50 % logic reuse and doubled development efficiency in production.
This article presents JuiceFS, an open‑source cloud‑native distributed file system that addresses the limitations of object storage for big‑data and AI workloads by providing strong consistency, high‑performance metadata, multi‑protocol support, small‑file management, and deep Kubernetes integration.
Apache Hudi is being re‑positioned from a simple table format to a full‑featured data lake platform, offering transactional storage, MVCC concurrency, metadata services, Deltastreamer ingestion, and plans for cache and timeline metadata services, aligning its vision with modern lakehouse architectures.
The talk by Beike senior engineer Yang Zongqiang details the evolution of the company's data development platform, covering background, three architecture upgrades, platform features such as metadata management, data integration, scheduling, quality assurance, and future directions for building an enterprise‑grade big‑data system.
This article describes how NetEase developed a Hive metadata migration and merging tool that consolidates metadata from multiple independent Hive clusters into a single Hive metastore without moving HDFS data, detailing the challenges, ID handling, database operations, and step‑by‑step migration process.
This article introduces Apache Iceberg as an open table format for massive analytic datasets, explains its underlying concepts such as schema, partitioning, statistics, and read/write APIs, compares it with Hive Metastore, outlines its ACID commit process, highlights the performance and operational advantages for big‑data workloads, and previews upcoming community features.
This article explains the concepts and practices of data modeling, data architecture, master data management, metadata management, and data quality management, describing their roles, key components, and how they support reliable and consistent enterprise information systems.
This article explains how a metadata‑driven multi‑tenant data model decouples logical and physical schemas, enabling rapid SaaS product rollout, seamless scaling, fine‑grained customization, and zero‑downtime schema changes while ensuring data isolation, security, and high performance across millions of tenants.
This article explains the fundamentals and value of metadata, describes a comprehensive metadata management system and its layered architecture, outlines key technologies such as automatic SQL metadata extraction, and showcases practical applications like metadata query, impact analysis, data lineage, and business‑driven data needs within modern data warehouses.
This article presents Hulu's multi‑layered approach to content understanding and embedding, describing tag‑based graph embeddings, metadata‑BERT enhancements, multimodal video/audio feature aggregation, and various applications such as similarity search, ranking, cold‑start retrieval, and collection modeling, while also discussing current limitations and open research questions.
The article provides a comprehensive overview of the Qirin big‑data platform, detailing its architecture, core modules such as resource management, metadata, data ingestion, task development, interactive query, and self‑service analysis, and outlines future development plans for the system.
This article introduces the development history, current architecture, abstraction methods, and key application scenarios of Kuaishou's metadata platform, highlighting challenges such as heterogeneous data integration, large-scale asset management, and the platform's role in data search, lineage, governance, and future enhancements.
The article introduces the Qirin big data platform—a one‑stop solution covering resource management, metadata, data ingestion, task development, interactive querying, and self‑service analysis—detailing its modular architecture, typical processing workflow, and future development plans for enterprise‑wide data services.
The platform implements comprehensive metadata governance by extracting, standardizing, and ingesting basic, trend, resource, lineage, and task metadata from offline and real‑time systems via a Kafka‑based SDK, enabling unified storage, monitoring, alerts, and future automation to improve data asset visibility and quality.
This article details iQIYI's transition from a fragmented Data Warehouse 1.0 to a unified, standardized Data Warehouse 2.0, covering layered architecture, dimension and metric design, modeling workflows, metadata management, data lineage, and upcoming intelligent and automated data platform initiatives.
This article presents a comprehensive overview of Hive SQL lineage parsing, detailing the challenges of data provenance in large‑scale data warehouses, introducing ANTLR‑based parsing techniques, and describing a series of optimizations—including AST pruning, CTE handling, UDF registration, and metadata service integration—to improve both table‑level and column‑level lineage extraction and visualization.
File systems risk data corruption during power loss or crashes because writes are not atomic, so journaling—recording intended operations in a write‑ahead log before committing them—ensures metadata and user data consistency, with variations like data journaling and ordered (metadata) journaling improving performance and reliability.
The article explains how Alibaba Cloud's Data Lake Building Service enables fine‑grained lake management by integrating multiple compute engines—including EMR, MaxCompute, Blink, Hologres, PAI, and open‑source Hive, Spark, and Presto—through unified metadata and OSS storage, while outlining current features, special format support, and planned future enhancements.
The talk chronicles iQIYI’s shift from a fragmented five‑layer Data Warehouse 1.0 to a unified 2.0 architecture featuring a central Dimension Layer, business‑focused data marts, and subject‑oriented warehouses, while detailing platform services, rigorous metadata management, lineage tracking, and future goals of intelligent, automated, service‑oriented, model‑driven data governance.
Journal file systems use write‑ahead logging to record each write operation as a transaction, ensuring that after power loss or crashes the system can replay logs and maintain metadata and user‑data consistency, avoiding corruption and space waste through techniques like data, ordered, and metadata journaling.
This article explains how Kafka relies on ZooKeeper to store cluster metadata, detailing the ZK node hierarchy, the process by which clients locate brokers, the broker‑side handling of metadata requests, and recommended practices for large‑scale deployments.
The Metadata Center, built by Alibaba Entertainment, streamlines development by offering a searchable Data Source Plaza and a configurable Custom Interface engine that abstracts service calls, adds unified monitoring and circuit‑breaker safeguards, and leverages optimized scripting, cutting upfront coding effort and accelerating feature delivery across Youku applications.
This article explains the fundamentals of object storage, compares it with block and file storage, outlines its architecture, components, advantages, use cases, and limitations, showing why it has become the dominant storage model in modern cloud environments.
This article presents a comprehensive guide to constructing real-time data warehouses using Apache Flink, covering the motivations, design principles, application scenarios, layer-by-layer architecture, metadata and lineage management, quality assurance, and the supporting toolchain for reliable streaming analytics.
This article describes the background, metadata concepts, SPU and category‑attribute modeling, MySQL tree‑storage techniques, dynamic business configuration, rule‑engine and workflow‑engine solutions adopted in Tripyun‑Ctrip Cloud to build a reusable, scalable middle‑platform for the tourism industry.
This article explains why metadata management is crucial for storage systems and reviews four typical approaches—initial external‑DB storage, in‑memory loading, partitioned services with a proxy layer, and tiered caching/persistence—illustrated with diagrams and real‑world examples.
The article explains Alluxio's tiered metadata management architecture, describing how the system separates hot and cold metadata into cached and persisted layers, and details the custom asynchronous eviction thread and cache implementation that replace Guava cache for efficient large‑scale metadata handling.
This article explains the concepts of data modeling and architecture, including logical data, process, and rule modeling, various data model types, master data management principles, metadata categories, and data quality management practices, highlighting their roles in enterprise information systems.
The article provides a comprehensive overview of banking data governance, covering core domains such as data models, metadata, standards, quality, lifecycle, distribution, exchange, security, and services, and explains how big‑data techniques can improve risk control, product innovation, and operational efficiency.
The article traces Youzan’s data warehouse from its chaotic early days lacking structure, through a 2016 Airflow‑driven construction phase that introduced layered ODS/DW/Data Mart architecture and naming standards, to a mature stage focused on efficiency, security, SparkSQL, dimensional modeling, metadata, and ongoing real‑time and governance challenges.
Meituan Delivery’s data‑governance framework combines a four‑layer warehouse architecture with comprehensive business, technical, security, and resource‑management standards, continuous metadata and security controls, and tools such as Wherehows and QuickSight, delivering standardized, secure, and easily shareable data while guiding future optimization and emerging‑technology adoption.
This article explains Unix/Linux inodes—the metadata structures that store file information—covering their purpose, contents, size considerations, inode numbers, directory handling, hard and soft links, and special inode-related operations, with practical command examples and visual illustrations.
The article surveys modern big‑data architecture, contrasting Lambda and Kappa models, highlights common governance and integration pain points, and proposes an end‑to‑end platform featuring unified metadata, stream‑batch processing, one‑click ingestion, standardized modeling, intelligent query abstraction, and a comprehensive development IDE.
This tutorial explains how to use Spring Boot's Configuration Processor to generate JSON metadata for configuration properties, covering dependency setup, Java bean definitions, property files, tests, and how the resulting metadata improves IDE auto‑completion and documentation.
This article provides a comprehensive guide to Kubernetes resource metadata, covering labels, annotations, ownerReferences, selector syntax, practical kubectl operations, and the underlying controller pattern that drives declarative API behavior in cloud‑native environments.
This tutorial explains Kubernetes resource metadata—including spec, status, labels, annotations, and ownerReferences—demonstrates how to query and modify them with kubectl, and dives into the controller pattern, control loops, and the advantages of declarative APIs over imperative ones.
The article explains the MFS distributed file system, detailing its four components—Master, Metalogger, Chunkserver, and Client—along with hardware recommendations, metadata handling, replication strategies, and FUSE‑based client mounting, providing a comprehensive guide to building a GoogleFS‑like storage cluster.
This article explains how to design a high‑performance feed‑stream architecture—including product definition, data modeling, storage choices, synchronization modes, metadata handling, commenting, likes, sorting, search, and deletion—so that a system can support tens of millions to billions of users while remaining reliable and scalable.
This article presents a comprehensive overview of Taobao's cloud computing architecture, detailing system components, various data synchronization methods such as TimeTunnel, Dbsync, and DataX, the scheduling system design, and metadata-driven analysis platforms for performance optimization and monitoring.
This article explains the evolution of MySQL's data dictionary from pre‑8.0 scattered metadata to the unified InnoDB dictionary tables in MySQL 8.0, covering storage structures, dictionary caching, information_schema changes, serialized dictionary information (SDI), atomic DDL mechanisms, persistent autoincrement handling, upgrade considerations, and provides practical code examples.
Journal file systems protect against data corruption caused by power loss or crashes by recording each write operation as a transaction in a dedicated log, then committing the changes only after the log is safely stored, enabling replay to restore consistency.
Dubbo Admin 0.1, a freshly refactored standalone project, replaces the old Webx backend with Spring Boot, adopts Vue and Vuetify for the UI, integrates Swagger, and introduces updated configuration, tag routing, application‑level service governance, and metadata‑driven testing to fully support Dubbo 2.7 features.
The article explains how distributed storage systems solve the fundamental problems of data sharding and redundancy by describing three sharding methods (hash, consistent‑hash, and range‑based), the criteria for choosing a shard key, the role of metadata servers, and consistency techniques such as leasing, all illustrated with concrete examples and code snippets.
This article outlines the journey of a financial services company from manual DBA tasks through tool‑assisted operations to a fully automated platform, detailing the platform’s technical stack, functional modules, metadata design principles, evolving SQL audit workflow, and future directions for intelligent database operations.
On September 15, Youzan’s big-data team and NetEase YouShu hosted a technical sharing titled “The Road to Data-Driven Operations,” where speakers explored the evolution of Youzan’s data warehouse metadata system, the architecture of its big-data development platform, and the application of functional programming in visual data analysis, highlighting current trends and future directions.
This article walks through a gray‑scale, controlled upgrade of Hive from version 0.13 to 2.1, covering metadata schema analysis, syntax compatibility, new Hive‑2.1 features, UDF adjustments, performance tweaks, and a step‑by‑step procedure to ensure stability and zero service interruption.
This talk details Baidu Waimai's end‑to‑end ETL design, covering demand sources, data flow patterns, multi‑stage system evolution, storage choices, scheduling architecture, configuration‑driven processing, quality monitoring, and how data lineage enables transparent, self‑service data delivery.
In this talk, a Baidu Waimai engineer explains the motivations, requirements, and architectural choices behind their open‑source ETL platform, covering data flow patterns, logical mappings, storage options, scheduling, metadata management, and quality monitoring to achieve scalable, transparent, and explainable data delivery.
In this talk, Hulu’s research director Zhou Hanning explains the key factors influencing recommendation system performance, describes optimization goals, explores collaborative filtering, matrix factorization, and neural‑network approaches—including metadata‑driven transfer learning and cold‑start solutions for live streaming—and shares practical AI implementations that improve user experience and engagement.
This article examines Ceph's Bluestore storage engine, describing its raw‑device management, metadata structures, write and read I/O processing, allocation strategies, and clone handling, highlighting how it reduces write amplification and optimizes for SSDs compared to the older Filestore.
This guide shows how to configure Vim so that pressing F4 automatically adds or updates a header containing author details, file name, and modification timestamp, using custom functions and a mapping, making it easy for programmers to embed metadata into their scripts.
The article explains Google’s internal Midas Package Manager (MPM), detailing its build definition files, immutable and mutable metadata stored in Bigtable, distributed replication via Colossus, client‑side pull and P2P copying, as well as its access‑control, encryption, and signing mechanisms that enable massive, conflict‑free software deployment at Google’s scale.
This article explains Oracle ASM metadata concepts, including ASM file types, allocation units, physical and virtual metadata structures, disk header composition, and practical methods for querying, validating, backing up, and restoring ASM metadata using tools like KFOD, KFED, and X$KFFXP.
This article examines the problems and trade‑offs of metadata protection in cloud object storage, highlighting the importance of reliability, availability, and consistency, and comparing primary‑secondary and multi‑replica models, including the WRN algorithm and practical operational considerations.
This article explains the concept of an inode, its role in storing file metadata, its size and structure on disk, how inode numbers are used, and the differences between hard links, soft links, and directory entries in Unix/Linux systems.
