Tagged articles
data lineage
95 articles · Page 1 of 1
The article argues that conventional data trust systems — focused on physical integrity like quality rules, lineage, and security — are insufficient for AI-driven decision-making; enterprises need a three-layer framework adding semantic trust (machine-readable business ontologies) and reasoning trust (confidence tags, circuit breakers) to align data with business logic for reliable AI reasoning.
AI transformationSemantic Layerbusiness ontology
0 likes · 12 min read

This article explains how to implement data lineage at three levels—table, field, and metric—detailing the purpose, methods, and limitations of each, and recommends a phased approach for enterprises to build traceable, assessable, and maintainable data governance.
ETLSQL parsingdata governance
0 likes · 19 min read

This document transforms the 'Business-Finance Single Account' design slides into an executable implementation blueprint for the OntoL ontology product, detailing a five-layer architecture, core object models, action schemas, semantic layer governance, budget control gates, simulation capabilities, and a four-phase rollout plan with acceptance criteria.
Data ModelingImplementation RoadmapSemantic Layer
0 likes · 47 min read

This article explains data lineage concepts, table-level vs field-level lineage, why major tech companies prioritize it, core use cases like impact analysis and root cause detection, three implementation approaches, common pitfalls, and a practical four-step guide for enterprises to start building lineage from critical reports.
ETLRoot Cause Analysisdata governance
0 likes · 18 min read

The article defines high-quality datasets in the AI era by six dimensions—accuracy, completeness, consistency, timeliness, understandability, and traceability—explains why AI amplifies data quality requirements, and outlines a five-step engineering pipeline (aggregation, cleaning, unification, validation, traceability) to transform raw data into reliable AI-ready datasets, emphasizing that continuous data engineering, not data hoarding, enables AI success.
AI data preparationData EngineeringFineDataLink
0 likes · 14 min read

This article dismantles twelve common data governance fallacies — from demanding instant metadata completion to treating governance as a cost center — arguing that effective governance requires confronting upstream data pollution, aligning incentives across departments, and prioritizing critical data paths over blanket coverage.
data governancedata lakedata lineage
0 likes · 10 min read

As data interfaces proliferate, quality issues shift from simple errors to semantic drift across systems; this article analyzes meaning, time, and responsibility drift, proposes a four-question lineage framework, and advocates lightweight change records to maintain trust in evolving data ecosystems.
GB/T 34960change managementdata governance
0 likes · 12 min read

This article explains the eight core modules of data governance—standards, metadata, catalog, quality, master data, assets, lineage, and security—detailing how each solves specific problems and why they must integrate into a continuous lifecycle loop rather than exist as isolated tools.
data assetsdata catalogdata governance
0 likes · 20 min read

Snowflake's new Data Lineage for Cortex Agents tracks agent data reachability through governed semantic views, extending lineage from tables to AI agents; Databricks pursues similar governance via Unity Catalog and Genie Ontology, positioning the semantic layer as a critical AI governance boundary for accuracy, authorization, consistency, and auditability.
AI governanceCortex AgentsDatabricks
0 likes · 16 min read

The article explains that metric management is not just cataloging metrics but governing their full lifecycle—business definitions, calculation rules, responsibility assignment, data lineage, and versioned change control—to ensure metrics are trustworthy, traceable, and sustainably governed across business, data, and technical layers.
FineDataLinkcalculation ruleschange management
0 likes · 19 min read

This article presents a comprehensive framework for building a big data governance platform, covering a five-layer architecture, nine core governance modules, a three-phase implementation roadmap, organizational structure, success metrics, and industry-specific selection guidance to transform chaotic data into trusted, findable, and controllable assets.
Implementation Roadmapbig data governancedata architecture
0 likes · 10 min read

The article examines the emergence of personal digital twins, outlines their five‑layer technical architecture, analyzes three ownership dilemmas—including raw data vs. model rights, cross‑platform portability, and liability for autonomous actions—and reviews UK, EU, and Chinese regulatory proposals along with practical enterprise solutions such as data lineage, exportable state snapshots, and audit‑driven circuit‑breakers.
AI agentsLoRAPersonal Digital Twin
0 likes · 12 min read

China's high-quality dataset count hit 120,000 totaling 1,565 PB with 60% quarterly growth, exposing four systemic gaps in traditional BI-oriented lakehouse architectures — storage, governance, compute performance, and security — and driving an AI-native reference architecture built on Paimon, StarRocks, tiered storage, operator-level lineage, and multi-modal retrieval.
AI training dataApache PaimonStarRocks
0 likes · 43 min read

The article explains how scattered, hard‑coded metadata in large‑scale systems leads to unknown impact of changes, and walks through building a unified metadata center with a single source of truth, automatic collection, lineage, impact analysis, asset discovery, active metadata, and governance, while highlighting practical trade‑offs and real‑world tools.
active metadatadata catalogdata governance
0 likes · 19 min read

This article argues that data sharing initiatives often fail not due to access permissions but because identical metric names mask divergent definitions, statistical scopes, and processing logic, a problem amplified in AI-driven workflows where semantic context must be explicitly preserved for reliable automated decisions.
AI governancecross-domain data circulationdata governance
0 likes · 12 min read

This article presents a five-layer evaluation framework for high-quality AI datasets—covering basic data quality, business semantics, task adaptation, AI application effects, and trustworthy operations—emphasizing task-specific validation over generic metrics and advocating admission vs. optimization metrics with automated, expert, and task-based verification.
AI data qualityRAG EvaluationTrustworthy AI
0 likes · 13 min read

This article outlines a six-step methodology for building high-quality AI datasets as semantic data products, starting from business scenario definition through semantic modeling, evidence mapping, task-specific processing, multi-dimensional validation, and continuous iteration via runtime feedback.
AI datasetsAgentRAG
0 likes · 12 min read

Enterprises often build many systems and reports, but without a systematic data‑asset management framework—comprising a data catalog, unified standards, quality controls, and lineage tracing—issues like inconsistent metrics, unknown data sources, and hard‑to‑diagnose report errors persist, undermining reliable decision‑making.
data asset managementdata catalogdata lineage
0 likes · 12 min read

Data modeling isn’t about creating more tables; the real challenge lies in translating business logic, relationships, and metric definitions into a stable structure, which requires mastering seven key concepts—business objects, granularity, entity relationships, fact and dimension tables, metric definitions, model layering, and data lineage and quality.
Data ModelingDimension TableFact Table
0 likes · 14 min read

The article explains how data lineage in Retrieval‑Augmented Generation systems links data quality, ingestion, and incremental sync into a traceable whole, detailing the five lineage nodes, schema trade‑offs, storage choices, and how lineage supports debugging, impact analysis, and version control.
RAGVector Databasedata governance
0 likes · 15 min read

This article details Qichacha's use of large language models to extract field‑level data lineage from heterogeneous, non‑standard code and ETL assets, describing the motivation, architectural blueprint, practical challenges such as cost, accuracy and hallucination, and the resulting improvements in impact analysis, metric tracing, and sensitive‑data governance.
FlinkLLMQichacha
0 likes · 11 min read

The article defines metadata as data about data, illustrates it with everyday examples, classifies it into technical, business, and management categories, and outlines eight core functions—including data asset mapping, fast search, flexible views, tagging, insight, lineage, impact analysis, and mapping—to improve data understanding, efficiency, quality, and cross‑system integration.
Data Managementbusiness metadatadata catalog
0 likes · 9 min read

This article details Qichacha's technical journey of applying large language models to resolve field‑level data lineage challenges in a complex, multi‑source data environment, describing the motivation, architecture, practical implementation, engineering trade‑offs, and measurable outcomes.
AIFlinkLLM
0 likes · 11 min read

This article explains the distinct meanings of data lineage, data quality, and data maps, illustrates their practical applications such as root‑cause tracing, compliance auditing, and asset discovery, and outlines common challenges and best‑practice steps for implementing each concept in data governance.
ETLdata governancedata integration
0 likes · 12 min read

Most data‑governance initiatives fail not because of lacking technology but because they add friction; the article explains how companies mistakenly focus on rules, platforms, and processes, and offers a step‑by‑step approach—identifying high‑value tables, minimal metadata, targeted quality rules, and fast issue diagnosis—to make governance truly useful.
big datadata governancedata lineage
0 likes · 29 min read

This article outlines a practical seven‑step framework for constructing data lineage—from defining clear goals and scoping requirements to designing architecture, collecting lineage, building a knowledge base, visualizing it, and establishing ongoing operations—so enterprises can turn messy data warehouses into trustworthy AI assets.
AI readinessETLdata governance
0 likes · 14 min read

The article explains why solid data governance—especially mastering the five categories of data lineage (logical, architectural, periodic, behavioral, and scenario)—is the first step for companies pursuing AI, detailing each type, its benefits, and practical implementation steps.
architectural lineagebehavioral lineagebig data
0 likes · 10 min read

When an AI‑generated conversion‑rate model is rejected by other teams due to mismatched data definitions, this article shows how to align measurement criteria by making data lineage and calculation scope explicit, using a three‑step cross‑domain mutual‑recognition protocol, checklists, and arbitration scripts to reduce internal friction.
AI model governancearbitration protocolcross-department collaboration
0 likes · 6 min read

The article analyzes why metadata management often stalls in enterprises—due to scattered, inactive, and business‑misaligned metadata—and outlines a step‑by‑step approach to define core asset, relationship, and semantic information, automate collection from data flows, and build a queryable metadata network.
data catalogdata governancedata integration
0 likes · 11 min read

In the era of AI and multi‑cloud, this article analyzes the core challenges of data governance—data silos, quality gaps, and compliance risks—and explains how Apache Gravitino’s unified metadata architecture together with OpenLineage’s standardized lineage model provide a scalable, automated solution for intelligent, real‑time data management.
Apache GravitinoOpenLineagebig data
0 likes · 15 min read

A minor field name change can cause downstream report failures, exposing the lack of a formal data lineage; this article defines data lineage, outlines its four core dimensions, compares granularity levels, and presents a step‑by‑step architecture and governance process for establishing a practical lineage system.
ETLdata architecturedata governance
0 likes · 9 min read

The article explains the distinct purposes, contents, usage scenarios, and target users of data catalogs and data dictionaries, illustrates their complementary relationship, and highlights practical challenges and a tool‑based approach for keeping them up to date.
data catalogdata dictionarydata discovery
0 likes · 9 min read

This talk explains how a data platform tackled massive, heterogeneous enterprise data by using large language models and prompt engineering to automatically extract field‑level lineage from SQL scripts, achieve over 80% coverage, and raise accuracy above 95%, dramatically cutting impact‑analysis time.
AI for data engineeringPrompt EngineeringSQL parsing
0 likes · 6 min read

This article explains what data integration really means, outlines its three core values, compares four mainstream integration patterns, details a five‑step implementation framework, highlights frequent pitfalls, and offers practical recommendations for enterprises of all sizes.
ETLdata architecturedata governance
0 likes · 11 min read

The article outlines a data governance framework for big data platforms that addresses enterprise data standards, quality, and metadata management, detailing three core objectives—standard implementation, quality analysis, and lineage mapping—while providing a PDF roadmap for practical deployment.
big datadata governancedata lineage
0 likes · 5 min read

The article explains data lineage as the full‑lifecycle relationships between data objects, outlines its ownership, multi‑source, traceability, and hierarchical traits, illustrates with school and banking examples, and details why lineage is crucial for compliance, impact analysis, security, migration, and self‑service, while reviewing collection techniques.
ETLbig datadata governance
0 likes · 11 min read

The article explains why treating data as an asset matters, distinguishes data asset management from traditional data governance, and outlines four practical steps—inventory, standardization, quality & security, and value realization—to turn raw data into a strategic business resource.
data asset managementdata catalogdata governance
0 likes · 10 min read

The article defines data lineage, illustrates its importance for rapid root‑cause analysis, impact assessment, compliance and asset management, and walks through a three‑step approach—collection, integration, validation—using concrete examples and a real‑world tool.
ETLdata lineagedata warehouse
0 likes · 9 min read

The article outlines a comprehensive data‑warehouse quality assurance framework covering data lineage, layer‑level testing (timeliness, completeness, accuracy), application‑level checks, downgrade and backup strategies, and future plans for automated data shape reporting and static SQL analysis.
SQLbig datadata lineage
0 likes · 15 min read

This article introduces Douyin Group’s Data Asset Management Platform, explaining how its focus on data assets rather than raw metadata enables a comprehensive, real‑time big‑data lineage system that supports search, AI‑driven discovery, and diverse application scenarios across the organization.
Data Asset PlatformDouyindata lineage
0 likes · 6 min read

This article introduces Douyin Group’s Data Asset Management Platform, explains its shift from traditional metadata to comprehensive data assets, and details the evolution, architecture, and applications of its full‑linkage data lineage, highlighting why building accurate, real‑time lineage is critical for quality, security, and cost efficiency.
Data Asset PlatformDouyindata lineage
0 likes · 5 min read

This article introduces Douyin Group’s comprehensive data asset management platform, explains why it emphasizes data assets over raw metadata, outlines its full‑linkage lineage capabilities, and presents practical insights on building, applying, and future‑proofing big data lineage within complex enterprise environments.
Douyinbig datadata asset management
0 likes · 5 min read

This article explains how the rapid rise of AI and large‑model technologies is driving a paradigm shift in data governance toward intelligent, automated, and real‑time collaboration, outlines the challenges of multi‑cloud environments, and demonstrates how Apache Gravitino and OpenLineage provide a unified metadata and lineage solution that improves data quality, compliance, and business agility.
Apache GravitinoOpenLineagebig data
0 likes · 12 min read

This article introduces Douyin Group’s one‑stop Data Asset Management Platform, explains why the company focuses on data assets rather than raw metadata, and details the evolution, architecture, applications, and future outlook of its comprehensive big‑data lineage system.
Douyinbig datadata asset management
0 likes · 5 min read

This article introduces Douyin Group’s Data Asset Management Platform, explaining its shift from traditional metadata to comprehensive data assets, detailing the evolution, architecture, and applications of its full‑link big data lineage, and offering strategic guidance for building effective lineage systems.
Data AssetDouyindata governance
0 likes · 5 min read

The article details Xiaohongshu’s Dataverse platform evolution into a Data+AI system, highlighting inefficiencies in algorithm and data‑science workflows, the introduction of an interactive notebook, comprehensive data lineage, AI‑coding assistance, and future DataAgent plans to automate data engineering tasks.
AI codingDataOpsdata lineage
0 likes · 21 min read

This article clarifies data governance as a management discipline focused on people, responsibility, and authority rather than just technical tools, outlining four key principles: governance is about rules not data, it requires binding data with accountability and permissions, it demands continuous operational processes, and its ultimate goal is enabling data-driven business value.
Data Managementaccess controlbusiness value
0 likes · 12 min read

This article details Unicom Digital’s metadata management practice on its integrated data platform, covering the strategic background of data, key challenges, award-winning capabilities, three-pronged solutions—automation, linking+, and AI—along with practical implementations, full‑chain lineage, data responsibility, lifecycle management, and future AI‑driven enhancements.
AIautomationbig data
0 likes · 18 min read

This article presents a detailed overview of Douyin Group's Data Asset Management Platform, focusing on the evolution, architecture, modeling, metrics, and application scenarios of its large‑scale data lineage system, and outlines future directions for full‑coverage, fine‑grained lineage capabilities.
big datadata asset managementdata lineage
0 likes · 17 min read

This article introduces Douyin Group’s end‑to‑end data asset management platform, explains the evolution and architecture of its large‑scale data lineage system, presents quality metrics and ecosystem components, and outlines practical applications and future directions for data governance, development, and security.
Data Asset PlatformDouyindata governance
0 likes · 16 min read

JD.com’s data‑governance framework combines a health‑score‑driven, automated platform that cross‑verifies audit logs, builds full‑link and operator‑level lineage, introduces standard fields, and optimizes resource mixing, task staggering, and cross‑datacenter scheduling, while targeting real‑time AI‑enhanced detection and full automation.
JD.comResource Optimizationaudit logs
0 likes · 15 min read

This article presents JD.com’s comprehensive big‑data governance experience, detailing the background and challenges, the automated governance platform and its core technologies such as audit logs and full‑link lineage, strategies for resource optimization, and the roadmap toward real‑time, intelligent, and fully automated data governance.
JD.comResource Optimizationautomation
0 likes · 14 min read

Huolala’s data team details the evolution of its metadata management platform—covering architecture, stages from early Hive‑ETL to real‑time field‑level lineage, AI‑driven smart search, cost‑governance mechanisms, and security classifications—showcasing practical solutions for data discoverability, efficiency, and protection at scale.
AI searchcost governancedata lineage
0 likes · 27 min read

Douyin Group’s data asset management platform introduces a systematic "manage, find, use" approach that unifies metadata collection, full‑coverage data lineage, and a suite of applications across development, governance, asset utilization, and security, while outlining its architecture, modeling, quality metrics, and future roadmap.
data governancedata lineagemetadata management
0 likes · 14 min read

This article provides a comprehensive overview of Douyin Group's data asset management platform, detailing the evolution, architecture, and applications of its large‑scale data lineage system, and discusses future directions for enhancing data quality, cost efficiency, and security across the organization.
data governancedata lineagemetadata management
0 likes · 15 min read

This article compiles concise overviews of recent e‑commerce research, covering real‑time online learning re‑ranking models, causal inference for user growth, full‑link data lineage, TikTok's data governance and attribution solutions, Volcano Engine's metric management, AI Agent applications on 1688, and XinXuan Group's live‑stream data architecture.
AICausal Inferencedata governance
0 likes · 5 min read

This article explains how to construct a full‑link data lineage platform for e‑commerce, detailing its architecture, quality metrics, and practical uses such as table migration, field‑level tracing, and automated metric decomposition to improve data governance and efficiency.
data governancedata lineagee-commerce
0 likes · 14 min read

This article explains how to construct a full‑chain data lineage system, covering its overall architecture, quality measurement framework, and application layer, and demonstrates practical use cases such as handling data growth, monitoring warehouse changes, accelerating development, ensuring consistency, and automating metric decomposition in real‑world business scenarios.
big datadata governancedata lineage
0 likes · 14 min read

This article explains why a comprehensive data asset catalog is essential for modern enterprises, outlines its core components such as inventory, metadata, data lineage, standards and access control, details step‑by‑step construction methods, and highlights key applications in governance, quality, compliance, architecture and valuation.
big datadata catalogdata governance
0 likes · 13 min read

This article explains how to construct a full‑chain data lineage system for e‑commerce, covering the concepts of data lineage, the design of a lineage foundation, quality measurement, application‑level lineage, and practical use cases such as table migration, field‑level tracing, and automated metric decomposition.
data lineagedata qualitye-commerce
0 likes · 12 min read

This article presents a comprehensive data lineage system (Data-Lineage) for big data platforms, addressing challenges in heterogeneous data sources, multiple execution engines, and complex dependencies through hook-based architecture and modular design.
SQL parsingbig data architecturedata lineage
0 likes · 12 min read

This article presents a comprehensive overview of Tencent Oula's data lineage system, detailing its background, goals, architecture, modular construction, key technologies such as graph databases and SQL parsing, and various internal application scenarios including data governance, cost insight, and baseline monitoring.
SQL parsingcost analysisdata lineage
0 likes · 20 min read

The article presents Tencent PCG's comprehensive data governance solution, detailing the challenges of massive, heterogeneous data, the four‑chapter framework covering governance overview, meta‑warehouse construction, an open asset‑scoring system, and a one‑stop governance workbench, and explains how lineage, scoring, and rule‑engine mechanisms enable cost‑effective, continuous data governance.
Asset Scoringbig datadata governance
0 likes · 14 min read


HomeTechNov 15, 2023 · Industry Insights
This article explains the challenges of data asset lineage in large data warehouses, presents a comprehensive approach using business‑level instrumentation, SQL interceptor plugins, and ETL script parsing to generate fine‑grained lineage graphs, and demonstrates measurable improvements in coverage and zombie‑table cleanup.
ETL parsingSQL interceptordata governance
0 likes · 18 min read

Bilibili's SQLScan is a static‑code analysis tool that parses Hive, Spark, Presto and Flink SQL via Antlr4, builds a unified AST, applies engine‑specific metadata plugins for rule enforcement, provides field‑lineage and cost‑analysis services, and has processed hundreds of thousands of daily queries, intercepting thousands of problematic statements to improve data quality and operational efficiency.
BilibiliDataOpsSQL static analysis
0 likes · 11 min read

Facing fragmented metadata, unclear ownership, and costly data duplication, Weimeng implemented a comprehensive data asset governance framework—covering metadata standards, lineage visualization, metric normalization, and cost management—to boost data quality, security, and business value across its new‑retail platform.
data governancedata lineagedata operations
0 likes · 15 min read

This article presents iQIYI’s comprehensive data link governance practice, covering the motivations, offline and real‑time pipeline governance strategies, monitoring mechanisms, data lineage, and exploratory work such as intelligent attribution and field‑level lineage to improve data accuracy, timeliness, and reliability.
data governancedata lineageiQIYI
0 likes · 11 min read


政采云技术Jun 15, 2023 · Big Data
This article discusses the practical implementation of extracting table and field lineage information via the Spline REST API, analyzing API call frequency, server load tolerance, and the strategy of re-parsing lineage only when job versions change to optimize performance.
Data EngineeringREST APISpline
0 likes · 5 min read

This article explains what data lineage is, its components, why it matters for data quality, security, and operational efficiency, and provides a comprehensive implementation guide covering open‑source tools, commercial platforms, custom builds, graph‑database modeling, automatic and manual lineage capture, visualization, analytics, and evaluation metrics.
ETLdata governancedata lineage
0 likes · 18 min read

In this interview, three data‑governance experts from Tencent, ByteDance, and Alibaba discuss the fundamental processes, core domains such as metadata, data lineage, metric systems, data quality and security, the main challenges they face, and emerging trends like DataOps, AI‑driven automation, and privacy‑preserving technologies.
DataOpsdata governancedata lineage
0 likes · 14 min read

This article explains how to derive field‑level data lineage for FlinkSQL by leveraging Apache Calcite, covering the Calcite framework, FlinkSQL execution stages, the three‑step parsing approach, core source code details, practical Insert/Join examples, and extensions for lookup joins and UDTFs.
Apache CalciteFlinkSQLRelNode
0 likes · 12 min read

This article details the user‑driven requirements, problem analysis, design choices, and technical implementation of a high‑performance, scalable data lineage visualization built with a React‑Canvas hybrid architecture to support tens of thousands of tables in a big‑data environment.
CanvasReactdata lineage
0 likes · 13 min read

This article presents an in‑depth overview of data lineage at ByteDance, detailing the design of storage, display, abstraction, implementation, and storage layers, optimization techniques for real‑time updates and queries, open export methods, practical use cases across asset, development, governance, and security domains, and future directions.
Apache AtlasJanusGraphdata lineage
0 likes · 20 min read

This article explains how effective metadata management provides context, improves data quality, enables data lineage tracing, supports governance, and ultimately turns raw data into valuable assets for enterprises navigating complex, evolving data environments.
Data ManagementEnterprise Datadata governance
0 likes · 35 min read

Data lineage in big‑data environments tracks how data moves and transforms—from source tables through SQL processing to final storage—enabling management tasks such as domain segmentation, performance tuning, anomaly detection, and dependency verification, with implementations ranging from simple regex extraction to robust AST parsing and optimization, as used by tools like Alibaba DataWorks and Apache Atlas.
ASTHiveSQL parsing
0 likes · 7 min read

This article explains how metadata serves as the foundation of enterprise data governance, outlines common data governance challenges, describes Wing Payment's metadata governance framework and platform architecture, and presents future directions such as multi‑source management, cross‑cluster disaster recovery, and intelligent recommendation.
big datadata governancedata lineage
0 likes · 18 min read

This article outlines Kuaishou's evolution of metadata management from its early Hive‑centric stage to a unified 2.0 platform, detailing system architecture, key technologies, challenges, and future 3.0 vision for low‑code, automated, and intelligent data governance.
big datadata discoverydata governance
0 likes · 15 min read

This article explains the growing importance of data lineage in large data warehouses, evaluates three Spark lineage extraction approaches, and provides a detailed, step‑by‑step guide to integrating the open‑source Spline agent—including codeless and programmatic initialization, configuration, dispatcher setup, post‑processing, and known limitations.
Apache SparkSparkSpline
0 likes · 16 min read

This article explains how to automatically extract data lineage and code dependencies from large collections of Python scripts by leveraging the language's compilation stages, abstract syntax trees, and the Pyflakes static‑analysis library, providing practical code examples and custom parsers for SQL extraction.
ASTCode Parsingbig data
0 likes · 12 min read

This article describes how Ctrip Finance built a large‑scale Nebula Graph platform for financial risk control, data lineage, and fraud detection, detailing the system architecture, real‑world applications, performance challenges, and the engineering optimizations applied to achieve sub‑15 ms query latency.
JavaNebula Graphdata lineage
0 likes · 18 min read

This article presents a request‑granularity quality model for Meituan's integrated recommendation system, linking data tables, algorithm models, services, and user requests, and details its metrics, defect taxonomy, calculation formulas, data‑lineage expansion, implementation, alert routing, and operational outcomes.
MeituanObservabilitydata lineage
0 likes · 22 min read

The article details how Bilibili built a unified metadata infrastructure—including a URN‑based model, collection pipelines, quality assurance, storage in TiDB/ES/HugeGraph, and query services—to support data discovery, lineage, impact analysis, and governance across its growing data platform.
ETLbig datadata catalog
0 likes · 21 min read

Bilibili’s data platform consolidates scattered metadata into a unified URN‑based model stored across TiDB, Elasticsearch, and HugeGraph, offering batch‑pull and embedded collection, flexible SQL‑like queries, comprehensive lineage mapping, and powering data‑map, lineage‑map, and impact‑analysis tools while planning expanded quality assurance and self‑service dictionaries.
SQL parsingdata governancedata lineage
0 likes · 21 min read

ByteDance’s Data Catalog article details the system’s unified metadata model, standardized ingestion connectors, search optimization techniques, lineage capabilities, and storage layer enhancements, highlighting key technical designs, performance improvements, and future work to advance data governance and asset utilization.
Search Optimizationdata catalogdata lineage
0 likes · 12 min read

This article explains ByteDance's data lineage architecture, covering data sources, processing pipelines, graph‑based modeling, key application scenarios, quality metrics such as accuracy, coverage and timeliness, and future directions for improving and standardizing lineage across its massive data platform.
Data Modelingbig datadata governance
0 likes · 14 min read

This article introduces the concept of a data map, explains its background and goals, describes Youzan’s end‑to‑end data‑map practice—including full data lineage, search, management, link analysis, impact estimation, and optimization—and concludes with a summary and future outlook.
Data ManagementFault Diagnosisbig data
0 likes · 16 min read

This article explains what data lineage is, why it is essential for data governance in large‑scale big‑data platforms, compares Apache Atlas with a custom solution, and details the technical choices, architecture, and performance optimizations behind the self‑built duo‑lineage system.
Apache AtlasSQL parsingbig data
0 likes · 14 min read

This article explains the fundamentals and value of metadata, describes a comprehensive metadata management system and its layered architecture, outlines key technologies such as automatic SQL metadata extraction, and showcases practical applications like metadata query, impact analysis, data lineage, and business‑driven data needs within modern data warehouses.
SQL parsingdata lineagedata warehouse
0 likes · 17 min read

This article introduces the development history, current architecture, abstraction methods, and key application scenarios of Kuaishou's metadata platform, highlighting challenges such as heterogeneous data integration, large-scale asset management, and the platform's role in data search, lineage, governance, and future enhancements.
Kuaishoudata lineagemetadata
0 likes · 16 min read

This article details iQIYI's transition from a fragmented Data Warehouse 1.0 to a unified, standardized Data Warehouse 2.0, covering layered architecture, dimension and metric design, modeling workflows, metadata management, data lineage, and upcoming intelligent and automated data platform initiatives.
Data Modelingdata lineagedata warehouse
0 likes · 25 min read

The talk chronicles iQIYI’s shift from a fragmented five‑layer Data Warehouse 1.0 to a unified 2.0 architecture featuring a central Dimension Layer, business‑focused data marts, and subject‑oriented warehouses, while detailing platform services, rigorous metadata management, lineage tracking, and future goals of intelligent, automated, service‑oriented, model‑driven data governance.
Data Modelingdata lineageiQIYI
0 likes · 23 min read

This article details the design, implementation, and optimization of a Flink‑based real‑time computing platform at Tongcheng‑Elong, covering the evolution from Storm to Flink, support for FlinkSQL and FlinkStream, metric collection, logging, data lineage, savepoint management, and numerous stability fixes contributed back to the open‑source community.
FlinkReal-time Computingbig data
0 likes · 16 min read

This article explains how Ele.me tackles big‑data challenges by designing a metadata governance platform that collects SQL execution data, parses lineage with Antlr, stores graph relationships in Neo4j, and enables table/column lineage queries, DAG scheduling, and hot‑data analysis.
Ele.meSQL parsingdata lineage
0 likes · 12 min read

This article examines the principles, features, and implementation details of distributed scheduling for big‑data ETL pipelines, covering decentralised schedulers, host selection strategies, fault‑tolerance, operator abstraction, elasticity, trigger mechanisms, visual monitoring, alarm handling, data fan‑in/fan‑out, parameter consistency, real‑time quality checks, lineage tracking, and field‑level traceability.
Distributed SchedulingETLHigh Availability
0 likes · 23 min read

Enterprise big‑data platforms face massive scale and complex metadata relationships, but using Transwarp Governor’s data lineage and impact analysis graphs enables precise tracing of data origins, rapid error localization, and prediction of downstream effects, dramatically improving data quality and governance efficiency.
Transwarp Governorbig datadata governance
0 likes · 8 min read
