Skip to content
Tagged articles

metadata management

130 articles · Page 2 of 2
Big Data Technology & Architecture
Big Data Technology & Architecture
Jun 29, 2021 · Big Data

Huawei Data Governance Practices and Metadata Management

This article outlines Huawei's data governance practices, detailing its digital transformation vision, two-stage data management evolution, structured and unstructured data classification frameworks, external data compliance, and comprehensive metadata management architecture, highlighting challenges and solutions for enterprise-wide data assets.

Huaweidata classificationdigital transformation
0 likes · 20 min read
Huawei Data Governance Practices and Metadata Management
Big Data Technology Architecture
Big Data Technology Architecture
Jun 10, 2021 · Big Data

Understanding Apache Iceberg: Design, Architecture, and Its Application at NetEase Cloud Music

This article explains Apache Iceberg’s table‑format design, compares it with Hive’s limitations, details its snapshot‑based architecture and metadata handling, and describes how NetEase Cloud Music leveraged Iceberg to dramatically improve large‑scale log processing performance and stability.

Apache IcebergSparkmetadata management
0 likes · 12 min read
Understanding Apache Iceberg: Design, Architecture, and Its Application at NetEase Cloud Music
macrozheng
macrozheng
May 8, 2021 · Big Data

Why Kafka 2.8 Drops Zookeeper: Architecture, Challenges, and KIP‑500

This article explains how Kafka 2.8 removes its dependency on Zookeeper, describes Kafka's core concepts and its interaction with Zookeeper, outlines the role of the Controller, discusses operational complexities and upgrade paths with KIP‑500, and highlights the benefits of the new KRaft‑based architecture.

KIP-500KRaftKafka
0 likes · 10 min read
Why Kafka 2.8 Drops Zookeeper: Architecture, Challenges, and KIP‑500
DataFunTalk
DataFunTalk
Feb 8, 2021 · Big Data

Ozone: The Next‑Generation Distributed Storage System Aiming to Replace HDFS

This article explains how Apache Ozone, built on the HDDS layer, addresses the scalability, memory, and performance limitations of HDFS by splitting metadata services, using RocksDB, implementing fine‑grained locking, RAFT‑based HA, and offering rich APIs, while outlining current challenges and future roadmap.

HDDSHDFSOzone
0 likes · 29 min read
Ozone: The Next‑Generation Distributed Storage System Aiming to Replace HDFS
DataFunSummit
DataFunSummit
Nov 17, 2020 · Big Data

Sohu Intelligent Media Data Warehouse Architecture and Technical Practices

This article presents Sohu Intelligent Media's data warehouse construction practice, covering fundamental concepts, batch and real‑time processing, OLAP theory, multidimensional modeling, workflow management, data quality, metadata lineage, and security, with a focus on Apache Doris and a Lambda‑style architecture.

Apache DorisOLAPbatch processing
0 likes · 18 min read
Sohu Intelligent Media Data Warehouse Architecture and Technical Practices
Beike Product & Technology
Beike Product & Technology
Nov 13, 2020 · Big Data

Beike One‑Stop Big Data Development Platform: Architecture, Evolution, and Future Outlook

The article summarizes Beike's one‑stop big data development platform, describing its data business background, the evolution from a simple Hadoop‑Kafka‑Hive stack to a metadata‑driven, asset‑oriented platform, and outlines current capabilities in data management, integration, scheduling, quality, openness, and future plans.

Data EngineeringETLbig data
0 likes · 11 min read
Beike One‑Stop Big Data Development Platform: Architecture, Evolution, and Future Outlook
Alibaba Cloud Developer
Alibaba Cloud Developer
Oct 25, 2020 · Big Data

How Alibaba’s Cloud‑Native Data Lake Solves Big Data Challenges

Alibaba Cloud’s Data Lake Analytics (DLA) tackles the growing complexity of data scenarios by offering cloud‑native, serverless solutions for data lake management, massive metadata construction, and high‑performance Spark and Presto engines, while addressing challenges such as high entry barriers, stability, and multi‑tenant isolation.

Serverless Sparkcloud-nativedata lake
0 likes · 22 min read
How Alibaba’s Cloud‑Native Data Lake Solves Big Data Challenges
ITPUB
ITPUB
Oct 16, 2020 · Big Data

How NetEase Cloud Music Built a Real‑Time Data Warehouse with Flink & Calcite

This article details NetEase Cloud Music's evolution of a real‑time data warehouse built on Flink 1.9 and Calcite, covering platform scale, architectural design, metadata management, SDK simplifications, monitoring improvements, and concrete use cases such as AB‑testing, live reporting, and feature serving.

CalciteFlinkbig data
0 likes · 8 min read
How NetEase Cloud Music Built a Real‑Time Data Warehouse with Flink & Calcite
Architecture Digest
Architecture Digest
Sep 12, 2020 · Backend Development

Zookeeper Usage Scenarios and Interview Analysis

This article explains common Zookeeper usage scenarios—including distributed coordination, distributed locking, metadata/configuration management, and high‑availability—provides interview‑style analysis, and illustrates each case with diagrams, helping Java developers understand how Zookeeper supports core distributed system functions.

High AvailabilityLockZooKeeper
0 likes · 5 min read
Zookeeper Usage Scenarios and Interview Analysis
58 Tech
58 Tech
Jul 13, 2020 · Big Data

Design and Implementation of a Financial Data Warehouse: Architecture, Modeling, Quality Monitoring, and Metadata Management

This article presents a comprehensive design and implementation guide for a financial data warehouse, covering background needs, modeling methodology choices, a layered architecture, data quality monitoring, metadata management, naming and coding standards, and future development directions.

Data ModelingETLFinancial Services
0 likes · 11 min read
Design and Implementation of a Financial Data Warehouse: Architecture, Modeling, Quality Monitoring, and Metadata Management
Big Data Technology Architecture
Big Data Technology Architecture
Apr 20, 2020 · Big Data

Introduction to HDFS: Architecture, Features, Replication, Rack Awareness, and Metadata Management

This article provides a comprehensive overview of Hadoop Distributed File System (HDFS), covering its streaming data access model, key characteristics, master‑slave architecture, block storage and replication mechanisms, rack‑aware placement strategy, and how the NameNode manages metadata and checkpoints.

HDFSHadoopRack Awareness
0 likes · 7 min read
Introduction to HDFS: Architecture, Features, Replication, Rack Awareness, and Metadata Management
Dada Group Technology
Dada Group Technology
Apr 15, 2020 · Big Data

Practice Experience of Dada Group's Real-Time Computation SQLization Using Dada Flink SQL

This article details Dada Group's development of the Dada Flink SQL engine, describing its background, architecture, parser design, dimension‑table join strategies, numerous enhancements such as HA support, Kafka keyword handling, metadata integration, Redis and ClickHouse sinks, BINLOG simplification, and future migration plans toward Flink 1.10.

ClickHouseFlinkReal-time Computing
0 likes · 12 min read
Practice Experience of Dada Group's Real-Time Computation SQLization Using Dada Flink SQL
dbaplus Community
dbaplus Community
Jan 14, 2020 · Big Data

How OPPO Built a Real‑Time Data Warehouse with Flink SQL

This article details{32-64 words} OPPO's evolution from an offline data warehouse to a real‑time platform, describing the business scale, data‑mid platform architecture, migration strategy using Flink SQL, extensions like AthenaX, and practical use cases such as real‑time ETL, CTR calculation, and tag import.

Data EngineeringETLFlink
0 likes · 18 min read
How OPPO Built a Real‑Time Data Warehouse with Flink SQL
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 12, 2019 · Big Data

Origin Data Governance Platform: Architecture, Modules, and Implementation at Meituan

The article describes Meituan's Origin Data Governance Platform, detailing its background, challenges, architectural redesign, core modules such as data storage, metadata, business, security, and application management, as well as its internal workflow, achievements, and future roadmap for unified, secure, and high‑performance data services.

Meituanmetadata managementplatform architecture
0 likes · 22 min read
Origin Data Governance Platform: Architecture, Modules, and Implementation at Meituan
Mafengwo Technology
Mafengwo Technology
Sep 26, 2019 · Big Data

Mafengwo’s Data Warehouse & Middle Platform: Architecture, Modeling, Toolchain

This article details Mafengwo’s journey in constructing a data warehouse and data middle platform, covering the core three‑layer architecture, hybrid modeling approaches, the supporting toolchain for data synchronization, scheduling, and metadata management, and the design of an indicator platform for business analytics.

Data ModelingETLbig data architecture
0 likes · 18 min read
Mafengwo’s Data Warehouse & Middle Platform: Architecture, Modeling, Toolchain
DataFunTalk
DataFunTalk
Aug 1, 2019 · Big Data

Streaming Data Platform Practices and Challenges at Beike Real Estate

This article presents an in‑depth overview of Beike's four‑layer streaming data platform, covering the foundational infrastructure, capability aggregation, data content, and output layers, as well as the challenges of metadata management, real‑time processing, and productization through the Ark and Tianyan systems.

Ark platformBeikeTianyan
0 likes · 14 min read
Streaming Data Platform Practices and Challenges at Beike Real Estate
Meituan Technology Team
Meituan Technology Team
Dec 27, 2018 · Big Data

Meituan Origin Data Governance Platform: Architecture and Practices

Meituan’s Origin Data Governance Platform inserts a unified governance layer between its data‑warehouse and application stacks, consolidating metric and dimension definitions, automating metadata management, enforcing security and workflow controls, and delivering cross‑engine query, monitoring and lineage capabilities that resolve inconsistencies and boost trust across dozens of internal data platforms.

metadata managementplatform architecture
0 likes · 21 min read
Meituan Origin Data Governance Platform: Architecture and Practices
Youzan Coder
Youzan Coder
Aug 3, 2018 · Big Data

Youzan Data Warehouse Metadata System: From Manual Tables to Metadata‑Driven Architecture

Youzan’s data‑warehouse metadata system evolved from manually maintained tables to an automated data dictionary and finally to a metadata‑driven architecture that automatically captures technical, business, and process metadata, visualizes lineage, tracks resource usage, manages synchronization rules and permissions, and now aims to improve novice usability with visual models and impact‑analysis tools.

HiveLineageResource Monitoring
0 likes · 11 min read
Youzan Data Warehouse Metadata System: From Manual Tables to Metadata‑Driven Architecture
ITPUB
ITPUB
May 30, 2018 · Backend Development

How JD.com Engineered Its Own Distributed Storage System for Billions of Files

This article chronicles JD.com's journey from recognizing massive storage demands to designing, building, and evolving a self‑developed distributed storage platform—JFS—that handles small and large files, powers a custom image system, object storage, and future container‑native workloads.

Backend EngineeringJFSObject Storage
0 likes · 16 min read
How JD.com Engineered Its Own Distributed Storage System for Billions of Files
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Jul 1, 2017 · Fundamentals

Designing Distributed File Systems: Solving Local FS Limits

Distributed file systems extend traditional local storage by partitioning data across multiple servers, using a master node for metadata and coordination, handling namespace, replication, load balancing, caching, and client interfaces, thereby overcoming file size, quantity, and concurrency constraints of ext3, reiserfs, and similar local filesystems.

CachingReplicationdistributed file system
0 likes · 15 min read
Designing Distributed File Systems: Solving Local FS Limits
ITFLY8 Architecture Home
ITFLY8 Architecture Home
May 8, 2017 · Fundamentals

Designing Scalable Distributed File Systems: Architecture, Challenges, and Solutions

This article explains how distributed file systems overcome the limitations of traditional local file systems by using a master‑metadata server, multiple data nodes, and client interfaces, and it details the key architectural components, common problems, and practical engineering solutions such as replication, load balancing, and caching.

Replicationarchitecturedistributed file system
0 likes · 15 min read
Designing Scalable Distributed File Systems: Architecture, Challenges, and Solutions
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Feb 23, 2017 · Fundamentals

How Ceph Monitor Uses Paxos to Ensure Consistent Metadata Management

This article explains the role of Ceph Monitor as the metadata management component in Ceph, detailing its centralized yet scalable design, the trade‑offs between centralized and peer‑to‑peer approaches, and how an improved Paxos algorithm with Bootstrap, Recovery, and read/write phases ensures consistent, fault‑tolerant cluster operation.

CephCluster ConsistencyPaxos
0 likes · 9 min read
How Ceph Monitor Uses Paxos to Ensure Consistent Metadata Management
Architects' Tech Alliance
Architects' Tech Alliance
Jan 6, 2016 · Fundamentals

Key Architectural Features and Technical Considerations of Flash Storage Systems

The article explains flash storage's low latency, high IOPS, and inline data protection features such as deduplication, compression, thin provisioning, while detailing scale‑out capabilities, symmetric A/A controller design, metadata management, global FTL functions, wear‑leveling, and power‑loss protection mechanisms.

SSDdeduplicationmetadata management
0 likes · 8 min read
Key Architectural Features and Technical Considerations of Flash Storage Systems
Architect
Architect
Nov 5, 2015 · Big Data

Design and Optimization of Large‑Scale Small File Storage Using Ceph and Merged Object Techniques

The article analyzes the challenges of storing massive numbers of small files, reviews existing solutions such as TFS and Haystack, and proposes a Ceph‑based approach that merges small files into RADOS objects with metadata stored as extended attributes, detailing the read/write/delete workflow and hash‑based placement strategies.

Cephmetadata managementobject merging
0 likes · 11 min read
Design and Optimization of Large‑Scale Small File Storage Using Ceph and Merged Object Techniques