Upgrading Data Warehouse Dependency Model: From Project-Level to Task-Level and External Dependency Integration

This article explains how a data warehouse dependency model was transformed from coarse project-level dependencies to fine-grained task-level DAGs, introduces virtual tasks for external dependencies, describes offset handling, and outlines the technical implementation and future automation plans for large‑scale scheduling systems.

DAGautomationdependency model

0 likes · 13 min read

Upgrading Data Warehouse Dependency Model: From Project-Level to Task-Level and External Dependency Integration

Bilibili Tech

May 12, 2023 · Big Data

Upgrade of Dependency Model in Bilibili Data Platform

Bilibili’s data platform upgraded its dependency model by shifting from project‑level to task‑level dependencies, adding root and end nodes, using virtual tasks for external data, introducing offset handling, implementing an abstract DependencySubject and asynchronous callbacks, achieving sub‑second latency for tens of thousands of daily tasks while planning automated lineage and richer rule support.

BilibiliData Platformdata dependency

0 likes · 10 min read

Upgrade of Dependency Model in Bilibili Data Platform