How Bilibili Uses LLMs to Tame Massive Data Platform Failures

Exploring Bilibili’s large‑scale data platform, this article details its five‑layer, storage‑compute separated architecture, the massive daily workload of offline and real‑time tasks, common failure and slowdown causes, and how an LLM‑powered intelligent assistant is being developed to help engineers troubleshoot efficiently.

BilibiliIntelligent AssistantLLM

0 likes · 5 min read

How Bilibili Uses LLMs to Tame Massive Data Platform Failures

DataFunSummit

Jun 19, 2025 · Big Data

How Shopee Leverages Paimon for Real‑Time Data Warehousing and Task Diagnosis

This article details Shopee's Data Infra team's use of the Paimon data lake to build near‑real‑time warehouses, accelerate ODS layers, implement a task‑diagnosis system, and create a reconciliation platform, while sharing future plans and a Q&A session.

Data LakeFlinkPaimon

0 likes · 12 min read

How Shopee Leverages Paimon for Real‑Time Data Warehousing and Task Diagnosis

DataFunTalk

Apr 4, 2023 · Big Data

Compass: An Open‑Source Big Data Task Diagnosis Platform for DolphinScheduler, Airflow and Spark

Compass is an open‑source big‑data diagnostic platform developed by OPPO that provides non‑intrusive, real‑time monitoring and root‑cause analysis for offline and streaming tasks on schedulers such as DolphinScheduler and Airflow, covering workflow‑level failures, Spark engine anomalies, resource usage, and offering one‑click reports and extensible rule‑based diagnostics.

Big DataDolphinSchedulerSpark

0 likes · 13 min read

Compass: An Open‑Source Big Data Task Diagnosis Platform for DolphinScheduler, Airflow and Spark