Tencent Cloud Developer
Aug 23, 2023 · Big Data
WeChat Experiment Platform: Architecture Design and Iceberg Lakehouse Optimization
The WeChat Experiment Platform migrated its 60,000 metric, 200,000 core, 30 PB plus data pipeline to an Iceberg based lakehouse, leveraging three layer metadata, fine grained partitioning, MERGE into writes, time travel snapshots and skew handling UDFs, which cut core time by 69%, saved ~100 PB storage, and reduced latency by up to 70%.
Data WarehouseMerge IntoMetric Computation
0 likes · 18 min read