Design and Implementation of a Scalable Data Lineage Graph in Volcano Engine DataLeap

Data lineage describes the source, transformation, and destination of data across processing stages and is essential for organizations to extract value from their data assets. Volcano Engine’s DataLeap platform provides a data‑map capability that visualizes table‑level lineage for thousands of tables, helping users discover dependencies and understand data flows.

Requirement discovery : Interviews with heavy internal users identified four primary scenarios—viewing table lineage, tracing upstream/downstream paths, grouping by key metrics, and filtering relevant information. Users need clear, efficient access to multi‑level dependencies and contextual table attributes.

Problem analysis : The previous lineage graph struggled with large data volumes, offering poor scalability, ambiguous node identification, unclear grouping, and inefficient filtering, which hindered usability for high‑volume environments.

Solution design : The new design expands node width to accommodate long table names, adopts a compact list‑based layout to preserve hierarchy while maximizing screen real‑estate, and introduces interactive features such as node highlighting, on‑hover task details, and customizable attribute displays.

Technical implementation : A hybrid React + Canvas approach was chosen. React renders interactive nodes, while Canvas draws performant connections. The initialization pipeline includes data preprocessing, node‑level calculation, grouping, layout, canvas setup, and rendering. Optimizations such as matrix‑based line updates, selective rendering of visible nodes, and server‑side filtering ensure smooth interaction even with tens of thousands of nodes.

Challenges and optimizations : Synchronizing Canvas and DOM refresh rates, reducing redraw overhead during scrolling, and limiting line rendering to visible connections dramatically improved frame rates. On‑demand node rendering and viewport‑aware line drawing further enhanced performance.

Overall, the revamped DataLeap lineage graph demonstrates how user‑centered design combined with thoughtful technology selection can deliver a clear, efficient, and scalable data lineage visualization for big‑data platforms.