How Bilibili Uses Large Language Models to Solve Big Data Task Failures

Background

Bilibili is a video‑sharing platform with massive data volumes; its big‑data platform supports many critical services such as AI and commerce.

The platform follows a "five‑layer integrated" plus "storage‑compute separation" architecture: a distributed file system at the bottom, an intelligent scheduling layer, various compute engines (Spark, Flink, etc.), client tools, and real‑time data streams (Kafka) and OLAP engines (ClickHouse), along with custom tools and CI/CD pipelines.

Every day the platform processes about 270,000 offline tasks, around 20,000 ad‑hoc queries, and roughly 7,000 important real‑time jobs. The support team receives thousands of inquiries weekly, requiring dedicated staff to handle about three person‑days of consulting per small team.

User Problems

For offline computation, users mainly ask two questions: why a task failed and why a task became slow.

Why tasks fail

Kernel defects – upgrades without sufficient testing can cause large‑scale failures.

Dependency component issues – upgrades or bugs in heavily used components break dependent tasks.

Data quality problems – corrupted or malformed data leads to failures.

Other reasons such as memory‑related errors.

Why tasks become slow

Hardware aging – massive storage devices wear out, reducing read/write speed.

Resource scheduling pressure – high user volume and mixed‑deployment mechanisms cause contention during peak periods.

Data distribution issues – data skew or problematic data sets degrade performance.

Diagnosing these failures or slowdowns is complex and time‑consuming, prompting the need for intelligent assistance.

Need for AI‑Driven Assistance

Users typically submit concise, engineering‑focused questions, often just a problem description with a link or screenshot. Automating the analysis of such queries with a large language model can help quickly identify root causes and suggest solutions.