The ByteDance Data Platform team empowers all ByteDance business lines by lowering data‑application barriers, aiming to build data‑driven intelligent enterprises, enable digital transformation across industries, and create greater social value. Internally it supports most ByteDance units; externally it delivers data‑intelligence products under the Volcano Engine brand to enterprise customers.
ByteDance’s Spark History Server was re‑engineered into a cloud‑native UIService that reduces storage usage by over 90%, cuts UI latency by up to 94%, and enables horizontal scaling, dramatically improving the user experience for large‑scale Spark jobs.
This article explains the background, design, and practical benefits of Hudi's Bucket Index—a hash‑based indexing mechanism that reduces unnecessary file reads and writes, improves upsert speed on terabyte‑scale datasets, and enables query optimizations such as bucket pruning and bucket join.
ByteDance’s EMR team details how they integrated data‑lake engines such as Hudi and Iceberg into SparkSQL, streamlined jar management, built a custom Spark SQL Server with Hive compatibility, multi‑tenant support, engine pre‑warming, and transaction capabilities, dramatically improving performance and resource efficiency for enterprise workloads.
This article examines how to design enterprise‑grade data warehouses by evaluating development convenience, ecosystem, decoupling, performance and security, compares Hive and SparkSQL along with other engines such as Presto, Doris and ClickHouse, and outlines best‑practice component selections for long‑running batch and interactive analytics.
ByteDance’s data platform front‑end team surveyed academic, commercial, and open‑source word‑cloud solutions, identified gaps in geo‑ and shape‑based clouds, and engineered a performant front‑end layout algorithm that generates customizable shape word clouds for diverse business scenarios.
This article surveys academic, commercial, and open‑source word‑cloud solutions, explains the underlying algorithms, visual encodings, layout strategies, interaction techniques, and classifications, and discusses the strengths, limitations, and future directions of word‑cloud visualisation.
This article explains how front‑end engineers can shift their mindset, adopt a BFF layer and GraphQL, and automate the integration with a data warehouse to become data owners, improve efficiency, and reduce reliance on back‑end resources in large‑scale data platforms.
This article details how ByteDance’s ClickHouse UBA edition improves dictionary encoding for low‑cardinality columns by redesigning the Part‑merge process, introducing a single‑dictionary merge, and implementing an automatic fallback for high‑cardinality columns, resulting in significant storage savings and query‑performance gains across large‑scale applications.
This article details ByteHouse’s evolution from ClickHouse, presenting two real‑time analytics use cases, the technical selection process, performance bottlenecks such as write throughput and Kafka consumption, and the engineered solutions—including asynchronous indexing, multi‑threaded Kafka engines, and enhanced Buffer engines—that enable reliable, high‑throughput data processing at massive scale.
This article explains why A/B testing is crucial for data‑driven product decisions, outlines ByteDance’s A/B testing system architecture across multiple layers, describes client‑ and server‑side experiment workflows, shares statistical best practices, and presents real‑world case studies illustrating hypothesis generation, evaluation, and future industry trends.
