Taobao Live’s Shift from ETL to Managed Development with DataWorks Data Agent

Traditional ETL pipelines faced a "human efficiency ceiling" and isolated expertise, making scaling difficult. Introducing large language models shifted the bottleneck: accuracy became the primary constraint, knowledge bases turned into essential infrastructure, and data consumption moved from static reports to direct AI calls.

Three‑Layer Architecture

Infrastructure Layer : Provides AI with sufficient input via a Common Dimensional Model (CDM), a knowledge base, and an engineering platform.

AI Capability Layer : Controls model behavior through Skill , Specification‑Driven Development (SDD), and AI‑assisted coding.

Application Layer : Enables effective usage via multi‑agent collaboration and both centralized and localized AI‑native execution.

The team chose NL2DSL2SQL instead of direct NL2SQL. By inserting a domain‑specific language (DSL) expressed in JSON between natural language and SQL, the pipeline gains:

Segmentable debugging

Pre‑audit validation

Reduced verification cost

When generated DSL violates platform rules, the system returns an error with precise location and triggers an AI self‑correction loop: 报错 → 定位 → AI 自修正 This intermediate DSL layer makes each logical step auditable and iteratable.

Data Agent Core

DataWorks Data Agent serves as the backbone, reinforced by two runtime anchors:

Ontology : Defines entities, relationships, and attributes (e.g., a global semantic for "anchor ID") to give AI an unambiguous knowledge reference.

Harness : Manages scheduling and context propagation so that long‑chain tasks retain session information without drift.

Accurate metadata—tables, fields, enums, code logic—is mandatory; otherwise AI‑generated code inherits probabilistic errors. The team standardized the CDM and enforced continuous asset freshness.

R&D Paradigm Reconstruction

The delivery process is split into two phases:

Clarification Phase : Build AI‑friendly technical solutions with manual checks.

Execution Phase : After a complete solution is input, AI drives the entire pipeline while humans perform only critical acceptance.

Six standardized steps emerged:

Abstract technical solution

Model construction with SQL templates

Fully automated code generation (no manual coding)

Online key‑information extraction (primary/foreign/distribution keys)

Automatic DDL changes

Monitoring via SOP + DQC

This forms an end‑to‑end automated pipeline from requirement focus to production delivery.

Multi‑agent collaboration is organized as a "digital factory": planning agents decompose requirements, functional agents execute tasks, and reporting agents aggregate results. Humans intervene only at high‑value decision points (data modeling, metric definition, complex business translation), while AI handles deterministic execution.

Skill, SDD, and AI Coding

Skill encapsulates reusable assets derived from the semantic layer and agent transformations, enabling zero‑cost reuse of development patterns.

Specification‑Driven Development (SDD) records requirements, resource gaps, model design, and I/O specifications, ensuring context preservation, full traceability, pre‑review, and templated execution.

AI coding follows two parallel paths:

DSL Path (NL → DSL → SQL) : Relies on the self‑built semantic layer, supports unit testing and engineering checks, and covers >70% of code volume.

DataWorks Data Agent Path : Lightly depends on the semantic layer, requiring only data sources, pseudo‑code, and CTE references; suitable for ad‑hoc queries and historical task modifications.

These paths raise code‑generation accuracy from roughly 50% to 80% and achieve near‑100% code‑generation penetration, delivering high‑quality results within 24 hours.

Infrastructure Foundations

OneData System : Extends beyond traditional data‑warehouse layering to provide precise, AI‑native delivery; all metadata must be accurate and continuously refreshed.

Knowledge Dual Engine : Combines LightRAG (graph‑based entity‑relationship network) to let AI locate tables, and a business wiki (terminology, metric definitions, enum dictionaries) to let AI understand domain language.

Engineering Platform : A self‑developed service platform that wraps semantic management, DSL validation, and Harness control as APIs, enabling dynamic orchestration in an AI‑native manner.

Future Evolution

Having automated the Requirement‑to‑Code (R2C) pipeline, the next focus shifts from faster coding to smarter data consumption. The team plans to launch ChatBI as a conversational interface that lets business users ask questions such as "Why did GMV fluctuate?" and receive AI‑driven attribution analysis, moving the paradigm from data development to data consumption.