Author

ByteDance Data Platform

The ByteDance Data Platform team empowers all ByteDance business lines by lowering data‑application barriers, aiming to build data‑driven intelligent enterprises, enable digital transformation across industries, and create greater social value. Internally it supports most ByteDance units; externally it delivers data‑intelligence products under the Volcano Engine brand to enterprise customers.

Articles

Likes

187

Views

Comments

Latest from ByteDance Data Platform

78 recent articles

ByteDance Data Platform

May 16, 2022 · Operations

How ByteDance’s SLA Assurance Platform Guarantees Data Reliability at Scale

This article explains how ByteDance’s self‑built SLA assurance platform addresses data pipeline communication costs, unclear responsibilities, and operational pressure by introducing roles, a streamlined signing workflow, checkpoint and recommendation calculations, and real‑time monitoring to achieve a 99.1% SLA compliance rate.

MonitoringSLAoperations

0 likes · 9 min read

How ByteDance’s SLA Assurance Platform Guarantees Data Reliability at Scale

ByteDance Data Platform

May 11, 2022 · Big Data

How to Build a High‑Performance SparkSQL Server with Hive JDBC Compatibility

This article explains how to design and implement a SparkSQL server that lowers usage barriers and boosts efficiency by supporting standard JDBC interfaces, integrating Hive Server2 protocols, handling multi‑tenant authentication, managing Spark job lifecycles, and providing high‑availability through Zookeeper coordination.

HiveJDBCServer Architecture

0 likes · 15 min read

How to Build a High‑Performance SparkSQL Server with Hive JDBC Compatibility

ByteDance Data Platform

Apr 27, 2022 · Big Data

How ByteDance Built a Scalable Data Catalog: Key Technologies and Future Plans

ByteDance’s Data Catalog article details the system’s unified metadata model, standardized ingestion connectors, search optimization techniques, lineage capabilities, and storage layer enhancements, highlighting key technical designs, performance improvements, and future work to advance data governance and asset utilization.

Data CatalogSearch Optimizationdata lineage

0 likes · 12 min read

How ByteDance Built a Scalable Data Catalog: Key Technologies and Future Plans

ByteDance Data Platform

Apr 20, 2022 · Operations

How Progressive Delivery Cuts Release Risk While Accelerating Iterations

This article explains progressive delivery and feature‑flag based smart release, showing how controlled rollout pace and stage authorization reduce deployment risk, enable rapid iteration, and integrate with CI/CD, A/B testing, and monitoring for safer software releases.

CI/CDProgressive Deliveryfeature flags

0 likes · 17 min read

How Progressive Delivery Cuts Release Risk While Accelerating Iterations

ByteDance Data Platform

Apr 15, 2022 · Cloud Native

How ByteHouse Evolved From ClickHouse Into a Next‑Gen Cloud‑Native Data Warehouse

ByteHouse, born from ByteDance’s extensive use of ClickHouse, transformed a high‑performance OLAP engine into a cloud‑native, scalable data warehouse by addressing scalability, elasticity, high availability, and multi‑tenant challenges through architectural redesign, custom storage layers, and advanced metadata management.

ByteHouseClickHouseData Warehouse

0 likes · 19 min read

How ByteHouse Evolved From ClickHouse Into a Next‑Gen Cloud‑Native Data Warehouse

ByteDance Data Platform

Apr 8, 2022 · Operations

How Baseline Monitoring Transforms Data Pipeline Reliability at ByteDance

This article explains ByteDance's baseline monitoring system for data pipelines, detailing its motivation, core concepts, architecture, instance generation, alert types, and handling of complex task dependencies to reduce operational costs and improve SLA compliance across hundreds of projects.

Data Pipelinesalertingbaseline monitoring

0 likes · 21 min read

How Baseline Monitoring Transforms Data Pipeline Reliability at ByteDance

ByteDance Data Platform

Mar 14, 2022 · Big Data

How ByteDance Cut Spark History Server Storage by 90% and Boost UI Speed

ByteDance’s Spark History Server was re‑engineered into a cloud‑native UIService that reduces storage usage by over 90%, cuts UI latency by up to 94%, and enables horizontal scaling, dramatically improving the user experience for large‑scale Spark jobs.

History ServerPerformance optimizationSpark

0 likes · 12 min read

How ByteDance Cut Spark History Server Storage by 90% and Boost UI Speed

ByteDance Data Platform

Feb 28, 2022 · Big Data

How Hudi’s New Bucket Index Boosts Upsert Performance in Massive Data Lakes

This article explains the background, design, and practical benefits of Hudi's Bucket Index—a hash‑based indexing mechanism that reduces unnecessary file reads and writes, improves upsert speed on terabyte‑scale datasets, and enables query optimizations such as bucket pruning and bucket join.

Bucket IndexHash IndexHudi

0 likes · 16 min read

How Hudi’s New Bucket Index Boosts Upsert Performance in Massive Data Lakes

ByteDance Data Platform

Feb 25, 2022 · Big Data

Optimizing SparkSQL: ByteDance EMR’s Data Lake Integration and Multi‑Tenant Server

ByteDance’s EMR team details how they integrated data‑lake engines such as Hudi and Iceberg into SparkSQL, streamlined jar management, built a custom Spark SQL Server with Hive compatibility, multi‑tenant support, engine pre‑warming, and transaction capabilities, dramatically improving performance and resource efficiency for enterprise workloads.

EMRHudiIceberg

0 likes · 11 min read

Optimizing SparkSQL: ByteDance EMR’s Data Lake Integration and Multi‑Tenant Server

ByteDance Data Platform

Feb 21, 2022 · Big Data

Choosing the Right Components for Enterprise Data Warehouses: Hive vs SparkSQL

This article examines how to design enterprise‑grade data warehouses by evaluating development convenience, ecosystem, decoupling, performance and security, compares Hive and SparkSQL along with other engines such as Presto, Doris and ClickHouse, and outlines best‑practice component selections for long‑running batch and interactive analytics.

ArchitectureData WarehouseETL

0 likes · 19 min read

Choosing the Right Components for Enterprise Data Warehouses: Hive vs SparkSQL