BestHub
Sign in
Tagged articles
7 articles
Page 1 of 1
Bilibili Tech
Bilibili Tech
Dec 17, 2024 · Big Data

Apache Gravitino: Metadata Management Practices and Production Experience at Bilibili

Bilibili adopted Apache Gravitino as a unified metadata platform that decouples consumers, consolidates schemas and Fileset‑based unstructured data across heterogeneous sources, cuts metadata and storage costs, resolves inconsistencies, boosts Hive Metastore performance, and enables features such as Iceberg branching and future AI‑centric governance.

Apache GravitinoBig DataFileset
0 likes · 20 min read
Apache Gravitino: Metadata Management Practices and Production Experience at Bilibili
DataFunSummit
DataFunSummit
Oct 17, 2024 · Big Data

Waggle Dance Based Metadata Solution at Tongcheng Travel: Architecture, Migration Strategies, and Future Outlook

This article presents Tongcheng Travel's metadata solution built on the open‑source Waggle Dance project, detailing the three‑layer architecture, challenges of a monolithic Hive Metastore, evaluated migration plans, federation implementation, migration workflow, and future directions for unified metadata governance.

Data MigrationFederationHive Metastore
0 likes · 11 min read
Waggle Dance Based Metadata Solution at Tongcheng Travel: Architecture, Migration Strategies, and Future Outlook
ITPUB
ITPUB
Sep 29, 2023 · Big Data

How Vivo Scaled Hive Metastore Using TiDB: A Deep Dive into Big Data Metadata

This article recounts Vivo’s journey to horizontally scale its Hive Metastore service by evaluating MySQL sharding, the open‑source Waggle‑Dance gateway, and ultimately selecting TiDB, detailing the migration process, configuration tweaks, performance benchmarks, encountered issues such as primary‑key conflicts, index choices, memory spikes, and the solutions implemented to ensure stable, high‑performance metadata storage for massive data volumes.

Big DataHive MetastorePerformance Optimization
0 likes · 22 min read
How Vivo Scaled Hive Metastore Using TiDB: A Deep Dive into Big Data Metadata
vivo Internet Technology
vivo Internet Technology
Sep 27, 2023 · Big Data

Horizontal Scaling of Hive Metastore Service at Vivo: Evaluation, TiDB Migration, and Lessons Learned

Vivo’s big‑data team horizontally scaled its Hive Metastore by evaluating MySQL sharding (Waggle‑Dance) against a TiDB migration, ultimately adopting TiDB, which after a synchronized cut‑over delivered ~15% faster queries, 80% DDL latency reduction, linear scaling, low resource use, and valuable operational lessons.

Big DataHive MetastoreSQL
0 likes · 19 min read
Horizontal Scaling of Hive Metastore Service at Vivo: Evaluation, TiDB Migration, and Lessons Learned
DataFunTalk
DataFunTalk
Mar 13, 2022 · Big Data

Tencent Data Lake Metadata Governance Practice and Architecture

This article presents Tencent's data lake metadata governance practice, covering data lake fundamentals, the 3+2 architecture of storage, compute and unified metadata, multi‑tenant design, the re‑implemented Hive Metastore for online catalog, performance optimizations, and offline data‑governance capabilities.

Big DataData LakeHive Metastore
0 likes · 18 min read
Tencent Data Lake Metadata Governance Practice and Architecture
Liulishuo Tech Team
Liulishuo Tech Team
Dec 31, 2020 · Big Data

Migrating a Petabyte-Scale Big Data Platform to Alibaba Cloud: Architecture, Challenges, and Lessons Learned

This article details the end‑to‑end migration of a petabyte‑scale big‑data platform to Alibaba Cloud, describing the DSS synchronization system, its integration with Hive Metastore and Airflow, the gray‑release strategy, data‑consistency validation using Presto, and key takeaways for future cloud migrations.

Big Data MigrationDSSHive Metastore
0 likes · 10 min read
Migrating a Petabyte-Scale Big Data Platform to Alibaba Cloud: Architecture, Challenges, and Lessons Learned