BestHub
Sign in
Tagged articles

metadata management

116 articles · Page 2 of 2
Dada Group Technology
Dada Group Technology
Apr 15, 2020 · Big Data

Practice Experience of Dada Group's Real-Time Computation SQLization Using Dada Flink SQL

This article details Dada Group's development of the Dada Flink SQL engine, describing its background, architecture, parser design, dimension‑table join strategies, numerous enhancements such as HA support, Kafka keyword handling, metadata integration, Redis and ClickHouse sinks, BINLOG simplification, and future migration plans toward Flink 1.10.

ClickHouseFlinkReal-Time Computing
0 likes · 12 min read
Practice Experience of Dada Group's Real-Time Computation SQLization Using Dada Flink SQL
dbaplus Community
dbaplus Community
Jan 14, 2020 · Big Data

How OPPO Built a Real‑Time Data Warehouse with Flink SQL

This article details{32-64 words} OPPO's evolution from an offline data warehouse to a real‑time platform, describing the business scale, data‑mid platform architecture, migration strategy using Flink SQL, extensions like AthenaX, and practical use cases such as real‑time ETL, CTR calculation, and tag import.

Data EngineeringETLFlink
0 likes · 18 min read
How OPPO Built a Real‑Time Data Warehouse with Flink SQL
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 12, 2019 · Big Data

Origin Data Governance Platform: Architecture, Modules, and Implementation at Meituan

The article describes Meituan's Origin Data Governance Platform, detailing its background, challenges, architectural redesign, core modules such as data storage, metadata, business, security, and application management, as well as its internal workflow, achievements, and future roadmap for unified, secure, and high‑performance data services.

Meituanmetadata managementplatform architecture
0 likes · 22 min read
Origin Data Governance Platform: Architecture, Modules, and Implementation at Meituan
Mafengwo Technology
Mafengwo Technology
Sep 26, 2019 · Big Data

Mafengwo’s Data Warehouse & Middle Platform: Architecture, Modeling, Toolchain

This article details Mafengwo’s journey in constructing a data warehouse and data middle platform, covering the core three‑layer architecture, hybrid modeling approaches, the supporting toolchain for data synchronization, scheduling, and metadata management, and the design of an indicator platform for business analytics.

Big Data ArchitectureData ModelingData Warehouse
0 likes · 18 min read
Mafengwo’s Data Warehouse & Middle Platform: Architecture, Modeling, Toolchain
DataFunTalk
DataFunTalk
Aug 1, 2019 · Big Data

Streaming Data Platform Practices and Challenges at Beike Real Estate

This article presents an in‑depth overview of Beike's four‑layer streaming data platform, covering the foundational infrastructure, capability aggregation, data content, and output layers, as well as the challenges of metadata management, real‑time processing, and productization through the Ark and Tianyan systems.

Ark platformBeikeBig Data Platform
0 likes · 14 min read
Streaming Data Platform Practices and Challenges at Beike Real Estate
Meituan Technology Team
Meituan Technology Team
Dec 27, 2018 · Big Data

Meituan Origin Data Governance Platform: Architecture and Practices

Meituan’s Origin Data Governance Platform inserts a unified governance layer between its data‑warehouse and application stacks, consolidating metric and dimension definitions, automating metadata management, enforcing security and workflow controls, and delivering cross‑engine query, monitoring and lineage capabilities that resolve inconsistencies and boost trust across dozens of internal data platforms.

metadata managementplatform architecture
0 likes · 21 min read
Meituan Origin Data Governance Platform: Architecture and Practices
Youzan Coder
Youzan Coder
Aug 3, 2018 · Big Data

Youzan Data Warehouse Metadata System: From Manual Tables to Metadata‑Driven Architecture

Youzan’s data‑warehouse metadata system evolved from manually maintained tables to an automated data dictionary and finally to a metadata‑driven architecture that automatically captures technical, business, and process metadata, visualizes lineage, tracks resource usage, manages synchronization rules and permissions, and now aims to improve novice usability with visual models and impact‑analysis tools.

Big DataData WarehouseHive
0 likes · 11 min read
Youzan Data Warehouse Metadata System: From Manual Tables to Metadata‑Driven Architecture
ITPUB
ITPUB
May 30, 2018 · Backend Development

How JD.com Engineered Its Own Distributed Storage System for Billions of Files

This article chronicles JD.com's journey from recognizing massive storage demands to designing, building, and evolving a self‑developed distributed storage platform—JFS—that handles small and large files, powers a custom image system, object storage, and future container‑native workloads.

JFSbackend engineeringdistributed storage
0 likes · 16 min read
How JD.com Engineered Its Own Distributed Storage System for Billions of Files
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Jul 1, 2017 · Fundamentals

Designing Distributed File Systems: Solving Local FS Limits

Distributed file systems extend traditional local storage by partitioning data across multiple servers, using a master node for metadata and coordination, handling namespace, replication, load balancing, caching, and client interfaces, thereby overcoming file size, quantity, and concurrency constraints of ext3, reiserfs, and similar local filesystems.

Distributed File Systemcachingmetadata management
0 likes · 15 min read
Designing Distributed File Systems: Solving Local FS Limits
ITFLY8 Architecture Home
ITFLY8 Architecture Home
May 8, 2017 · Fundamentals

Designing Scalable Distributed File Systems: Architecture, Challenges, and Solutions

This article explains how distributed file systems overcome the limitations of traditional local file systems by using a master‑metadata server, multiple data nodes, and client interfaces, and it details the key architectural components, common problems, and practical engineering solutions such as replication, load balancing, and caching.

Distributed File Systemarchitecturemetadata management
0 likes · 15 min read
Designing Scalable Distributed File Systems: Architecture, Challenges, and Solutions
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Feb 23, 2017 · Fundamentals

How Ceph Monitor Uses Paxos to Ensure Consistent Metadata Management

This article explains the role of Ceph Monitor as the metadata management component in Ceph, detailing its centralized yet scalable design, the trade‑offs between centralized and peer‑to‑peer approaches, and how an improved Paxos algorithm with Bootstrap, Recovery, and read/write phases ensures consistent, fault‑tolerant cluster operation.

CephCluster ConsistencyPaxos
0 likes · 9 min read
How Ceph Monitor Uses Paxos to Ensure Consistent Metadata Management
Architects' Tech Alliance
Architects' Tech Alliance
Jan 6, 2016 · Fundamentals

Key Architectural Features and Technical Considerations of Flash Storage Systems

The article explains flash storage's low latency, high IOPS, and inline data protection features such as deduplication, compression, thin provisioning, while detailing scale‑out capabilities, symmetric A/A controller design, metadata management, global FTL functions, wear‑leveling, and power‑loss protection mechanisms.

SSDdeduplicationmetadata management
0 likes · 8 min read
Key Architectural Features and Technical Considerations of Flash Storage Systems
Architect
Architect
Nov 5, 2015 · Big Data

Design and Optimization of Large‑Scale Small File Storage Using Ceph and Merged Object Techniques

The article analyzes the challenges of storing massive numbers of small files, reviews existing solutions such as TFS and Haystack, and proposes a Ceph‑based approach that merges small files into RADOS objects with metadata stored as extended attributes, detailing the read/write/delete workflow and hash‑based placement strategies.

Cephmetadata managementobject merging
0 likes · 11 min read
Design and Optimization of Large‑Scale Small File Storage Using Ceph and Merged Object Techniques