Tagged articles
metadata service
19 articles · Page 1 of 1
This article analyzes the challenges of scaling ClickHouse within Baidu’s MEG data platform and details a lake‑house solution that decouples storage and compute, integrates a meta‑service for transparent data access, optimizes query performance through caching, data roll‑up and layout tuning, and introduces a unified query gateway that gracefully falls back to Spark for complex workloads.
ClickHouseQuery GatewaySpark
0 likes · 25 min read

This article explains InfiniFS's speculative path resolution, detailing how predictable directory IDs and parallel lookups transform traditional linear RPC-based path traversal into constant‑time operations, dramatically reducing metadata access latency in large, deep directory trees.
InfiniFSdistributed file systemmetadata service
0 likes · 8 min read

This article summarizes the InfiniFS paper, detailing how its access‑content decoupling, speculative path resolution, and optimistic metadata caching enable efficient metadata handling for data‑center‑scale file systems supporting billions of files.
InfiniFSdistributed file systemsfilesystem design
0 likes · 15 min read

This article dissects DeepSeek's 3FS parallel file system, detailing its four‑component architecture, high‑throughput RDMA networking, metadata handling with FoundationDB, client access methods, chain replication (CRAQ), custom FFRecord format, and recovery mechanisms, offering a deep technical perspective for storage engineers.
High-performance storageRDMAchain replication
0 likes · 22 min read

This article dives deep into DeepSeek's 3FS parallel file system, detailing its four-component architecture, RDMA‑based high‑speed networking, client options, metadata and storage services, replication protocols, dynamic stripe sizing, and recovery mechanisms that enable efficient AI model training and inference.
AI TrainingRDMAdistributed file system
0 likes · 21 min read

The article provides a comprehensive English overview of 3FS, an AI‑focused distributed file system that leverages FoundationDB for metadata, CRAQ for chunk replication, and a hybrid Fuse/native client architecture, detailing its design, components, fault handling, and performance considerations for large‑scale training workloads.
AI storageCRAQ replicationFoundationDB
0 likes · 25 min read

Facing petabyte‑level data, billions of small files, and the need for POSIX, S3, and HDFS compatibility, Didi designed a new generation of non‑structured storage—OrangeFS—by analyzing internal systems, combining multiple storage solutions, reusing GIFT technology, and implementing a high‑performance metadata service, multi‑protocol fusion, and robust scalability features.
AI storageMulti-Protocolbig data
0 likes · 27 min read

Vivo’s big‑data team horizontally scaled its Hive Metastore by evaluating MySQL sharding (Waggle‑Dance) against a TiDB migration, ultimately adopting TiDB, which after a synchronized cut‑over delivered ~15% faster queries, 80% DDL latency reduction, linear scaling, low resource use, and valuable operational lessons.
Hive MetastoreSQLTiDB
0 likes · 19 min read

REDck is a cloud‑native, storage‑compute separated real‑time OLAP data warehouse derived from ClickHouse that addresses scalability, operational cost, and reliability challenges through a unified metadata service, object‑storage optimizations, multi‑level caching, distributed task scheduling, and two‑phase commit transactions.
ClickHouseReal-time OLAPStorage-Compute Separation
0 likes · 18 min read

The talk from DataFun Summit 2023 explains how Baidu's CFS storage builds a trillion‑file‑scale distributed file system by revisiting file system fundamentals, POSIX limitations, historical storage architectures, and introducing a lock‑free metadata service with single‑shard primitives, data‑layout optimizations, and a simplified client‑centric architecture that achieves high scalability and performance.
CFSPOSIXcloud storage
0 likes · 31 min read


ITPUBSep 11, 2023 · Cloud Native
Xiaohongshu built REDck, a cloud‑native, storage‑compute separated real‑time OLAP warehouse on ClickHouse, addressing scaling, cost, and reliability challenges through a unified metadata service, object‑storage optimizations, multi‑level caching, distributed task scheduling, bucketing, and exactly‑once transaction support.
CachingClickHouseObject Storage
0 likes · 21 min read

This article explains Baidu's CFS architecture for building a billion‑file‑scale distributed file system, covering basic file system concepts, POSIX limitations, metadata service modeling, performance metrics, evolution of metadata architectures, and CFS's lock‑free design that achieves high scalability, low latency, and balanced load in cloud storage.
cloud storagedistributed file systemlock-free design
0 likes · 32 min read

Baidu’s CFS metadata service scales to billions of files by shrinking critical sections through a lock‑free Namespace 2.0 design that confines conflicts to single shards, uses field‑level atomic primitives, and integrates the proxy into the client, delivering up to 76× throughput gains and significant latency reductions in production.
Baidu CFSEuroSys 2023POSIX compatibility
0 likes · 40 min read

This article explains the design and evolution of Baidu Cloud File System's (CFS) metadata service, detailing how a novel lock‑free architecture and strategic data layout enable POSIX‑compatible, highly scalable storage that can handle billions of files while maintaining high performance and consistency.
cloud storagedistributed file systemlock-free
0 likes · 42 min read

The EuroSys 2023 paper on Baidu’s cloud‑native file storage CFS reveals a metadata service design that trims critical sections, achieving 1.76‑75.82× higher throughput and up to 91.71% lower latency compared to HopsFS and InfiniFS, and has been production‑stable for over three years.
EuroSyscloud native storagedistributed file system
0 likes · 6 min read

UCloud’s US3 object storage metadata service, originally built on a chained MongoDB architecture, faced scalability, performance, and cost challenges, prompting a redesign that introduces a high‑compatibility DB‑Gateway, a distributed KV store (UKV) with custom RocksDB, delivering faster reads, zero list‑service latency, 80% cost reduction, and simpler operations.
Cloud ArchitectureDistributed KVObject Storage
0 likes · 8 min read

The article explains how the cloud‑construction platform evaluated graph‑database options based on open‑source, scalability, latency, storage capacity and import capabilities, ultimately choosing NebulaGraph, and then details NebulaGraph’s distributed meta, storage and query services as well as the overall multi‑layer knowledge‑graph platform architecture and future application scenarios.
NebulaGraphQuery Servicedistributed architecture
0 likes · 11 min read

This article explains how Dubbo integrates with Kubernetes to provide application‑level service discovery using a metadata service and revision mechanism, detailing both API‑client and DNS‑client implementations, their workflows, advantages, drawbacks, and future directions toward cloud‑native adoption.
API clientDNSDubbo
0 likes · 12 min read

This article explains how eBay’s Sherlock.IO monitoring platform processes billions of logs, events, and metrics daily using Flink Streaming jobs, detailing a metadata‑driven architecture, shared job strategies, Heartbeat‑based monitoring, job isolation, back‑pressure handling, and real‑world use cases such as Event Alerting, Eventzon, and Netmon.
Flinkbig datametadata service
0 likes · 18 min read
