This talk explains how a data platform tackled massive, heterogeneous enterprise data by using large language models and prompt engineering to automatically extract field‑level lineage from SQL scripts, achieve over 80% coverage, and raise accuracy above 95%, dramatically cutting impact‑analysis time.
This article examines the security challenges of data export work orders in MySQL environments, outlines the shortcomings of the original 1.0 workflow, and presents a comprehensive 2.0 redesign that introduces dynamic approvals, data classification, execution‑plan analysis, and code‑level solutions to mitigate data leakage risks.
This article presents a comprehensive data lineage system (Data-Lineage) for big data platforms, addressing challenges in heterogeneous data sources, multiple execution engines, and complex dependencies through hook-based architecture and modular design.
This article details the design, architecture, modular implementation, and practical use cases of Tencent Oula's data lineage platform, covering background goals, system components, graph‑based algorithms, SQL parsing, cost‑allocation insights, and a Q&A session for data engineers and analysts.
This article presents a comprehensive overview of Tencent Oula's data lineage system, detailing its background, goals, architecture, modular construction, key technologies such as graph databases and SQL parsing, and various internal application scenarios including data governance, cost insight, and baseline monitoring.
This article explains how MySQL establishes connections, the supported communication protocols and message formats, the lexical and syntactic parsing process, query optimization and execution plan generation, the role of different storage engines, and how the execution engine uses the plan to operate on the storage layer.
The article outlines a data‑space architecture that employs a wide‑table design with dynamic columns and dedicated metadata tables, a metadata execution engine for business‑logic mapping, upgraded SQL parsing via Druid, MySQL‑proxy protocol handling, and distributed flow control using Redis and Zookeeper to enable scalable, multi‑tenant, low‑code and cloud‑native data management.
This article explains how to derive field‑level data lineage for FlinkSQL by leveraging Apache Calcite, covering the Calcite framework, FlinkSQL execution stages, the three‑step parsing approach, core source code details, practical Insert/Join examples, and extensions for lookup joins and UDTFs.
This article explains the complete MySQL execution flow, covering the connector, permission verification, caching, parser, optimizer, executor, process states, SQL execution order, and the impact of WHERE‑clause condition ordering on query performance.
This guide explains common business scenarios that require custom SQL parsing, walks through setting up Antlr4 in IntelliJ IDEA, configuring Maven dependencies, generating parser code, and provides Java examples for extracting table names from Spark SQL statements, including handling of prediction modes and execution results.
Data lineage in big‑data environments tracks how data moves and transforms—from source tables through SQL processing to final storage—enabling management tasks such as domain segmentation, performance tuning, anomaly detection, and dependency verification, with implementations ranging from simple regex extraction to robust AST parsing and optimization, as used by tools like Alibaba DataWorks and Apache Atlas.
This article introduces Apache Calcite as a modular data‑management framework, explains its architecture and SQL processing pipeline—from parsing and validation to relational‑algebra conversion, optimization, and execution—and demonstrates a complete CSV‑based query example with code snippets.
The third article in the Presto SQL Engine series demonstrates how to implement WHERE‑clause filtering with Antlr, contrasting a direct AST‑traversal visitor approach—hampered by branch prediction and JVM inlining issues—with runtime bytecode generation using airlift.bytecode, which yields roughly three‑fold speed gains but adds complexity.
Bilibili’s data platform consolidates scattered metadata into a unified URN‑based model stored across TiDB, Elasticsearch, and HugeGraph, offering batch‑pull and embedded collection, flexible SQL‑like queries, comprehensive lineage mapping, and powering data‑map, lineage‑map, and impact‑analysis tools while planning expanded quality assurance and self‑service dictionaries.
This article describes practical improvements to the my2sql binlog‑parsing tool—adding query support and automatic binlog range detection—and demonstrates how to integrate it into DBeaver to provide developers with easy data‑rollback and original‑SQL tracing without requiring DBA intervention.
This article explains what data lineage is, why it is essential for data governance in large‑scale big‑data platforms, compares Apache Atlas with a custom solution, and details the technical choices, architecture, and performance optimizations behind the self‑built duo‑lineage system.
This article explains the fundamentals and value of metadata, describes a comprehensive metadata management system and its layered architecture, outlines key technologies such as automatic SQL metadata extraction, and showcases practical applications like metadata query, impact analysis, data lineage, and business‑driven data needs within modern data warehouses.
The article describes how Liulishuo automates data‑warehouse task scheduling by parsing SQL to build DAGs, introduces systematic governance to improve reliability, reduce execution time, and handle unreliable data sources, ultimately enhancing data‑engineering efficiency.
This article explains how MySQL parses SQL statements using Yacc, detailing the grammar structure, definition and rule sections, relevant source files, parse‑tree node declarations, common error messages, and the call stacks involved in both parsing and pre‑parsing phases.
The article examines a severe database performance slowdown caused by excessive library cache locks, traces it to improper JDBC ResultSetType settings that introduce unwanted ROWID columns during SQL parsing, and presents systematic experiments across various queries, ResultSetType values, JVM configurations, and database versions to recommend optimal parameter choices.
This tutorial explains how to implement a simple LL(1) parser in Go for SQL queries, covering lexical analysis, syntax analysis, finite‑state‑machine strategy, and testing, providing complete code snippets and practical guidance for developers interested in parser construction.
This article explains how Ele.me tackles big‑data challenges by designing a metadata governance platform that collects SQL execution data, parses lineage with Antlr, stores graph relationships in Neo4j, and enables table/column lineage queries, DAG scheduling, and hot‑data analysis.
This article explains why protecting database systems is critical, reviews existing SQL‑analysis tools, and dives deep into MySQL's lexical and syntax parsing techniques—including Bison‑generated parsers, core data structures, and practical applications such as useless‑condition removal and SQL feature generation—to help DBAs automate and optimize their workflows.
The article explains how to extend MySQL’s built‑in lexer and Bison‑based parser to expose table names, query features, and optimization advice via a simple language‑agnostic service, illustrating core data structures, useless‑condition elimination, feature generation for slow‑query analysis, and practical learning tips.
The article explains how to generate and interpret Oracle Automatic Workload Repository (AWR) reports, detailing key sections such as DB Time, Cache Sizes, Load Profile, parsing behavior, instance efficiency percentages, shared pool statistics, and top wait events to diagnose database performance issues.
The article outlines UDDB’s roadmap from a MySQL‑compatible middleware to a full‑featured distributed database, detailing its three‑stage evolution, system architecture, SQL parsing and routing design, and innovative techniques that enable read‑write separation, vertical sharding, and future horizontal scaling.
This article introduces the architecture, key modules, and implementation details of the Little Boy platform, a Greenplum optimization and operations system that parses SQL, applies index and distribution‑key tuning, manages resources, and outlines future enhancements for large‑scale data warehouses.
This article explains the theoretical foundations of relational databases, outlines the essential storage, engine, and UI layers, and walks through a concrete minimal implementation using fixed‑length tables, B+‑tree indexes, simple SQL parsing with regular expressions, and a TCP‑based client interface.
This article breaks down Oracle's SQL parsing workflow into four stages—syntax check, semantic check, parse phase, and execution phase—detailing how hard and soft parses differ, why cursors may not be shared, and common reasons for parsing failures.
