How DataWorks Data Agent Evolved Across Three Stages and Its Cloud‑Native Engineering Practices

Evolution Stages of DataWorks Data Agent

Since 2023, DataWorks has advanced its intelligent capabilities through three incremental stages. The first stage, Copilot mode , provides assistance only—SQL completion and generation—while humans remain the primary operators; observed efficiency gains are about 30‑35% in code writing. The second stage, human‑AI collaboration , is the current phase, delivering 30‑100% efficiency improvements as the Agent gradually replaces traditional GUI operations. The third stage envisions AI‑driven autonomy , where humans merely assign tasks and the AI orchestrates execution, potentially achieving ten‑ to hundred‑fold efficiency gains.

Product Matrix: Four Types of Agents Covering the Full Data Lifecycle

DataWorks Data Agent is not a single function but a layered service. The model layer offers Qwen series, GLM series, and fine‑tuned expert models for NL2SQL, with optional self‑hosted models. The agent layer provides four agents:

Data engineering (ETL development)

Data governance

Data analysis (Chat BI)

Cluster control and operations optimization

The interaction layer supports multiple UI modalities.

Scenario 1: End‑to‑End Automation in Data Development

Traditional ETL relies heavily on manual coding, leading to long cycles, low efficiency, and high onboarding barriers. Data Agent enables demand‑driven, full‑link automation: users describe requirements in Markdown, and the Agent autonomously plans tasks, generates code, configures scheduling, performs validation, and publishes the job. Real‑world example : in a live‑order data‑warehouse project, the Agent automatically planned the ODS→DWD→DWS→ADS pipeline, applied appropriate engines for each layer, and adhered to lake‑warehouse standards, requiring only a brief human review before release.

Scenario 2: From Passive to Proactive Operations Diagnosis

Data operations traditionally react to alerts, often waking engineers at night. Data Agent’s diagnostic capability spans the entire task lifecycle—whether a task has not run, is running, or has failed. It performs multi‑dimensional correlation analysis, produces a structured diagnostic report, and offers remediation suggestions with one‑click fixes, all via natural‑language dialogue.

Scenario 3: Autonomous Governance Loop

Conventional data governance suffers from discoverability, manual rule definition, fragmented processes, and lack of continuous improvement. Data Agent reshapes the paradigm:

Intelligent asset discovery : users describe metric logic in natural language; the Agent retrieves relevant tables, generates a full‑lineage graph, and visualizes upstream/downstream relationships.

Automatic quality rule generation : based on a governance request, the Agent identifies target ODS tables, creates quality monitoring rules (e.g., row count variance, key format checks), and configures thresholds and alerts.

Goal‑driven autonomous governance : after setting a governance goal, the Agent analyses the warehouse, plans a governance roadmap, and executes it periodically, turning one‑off governance into a continuous closed‑loop.

Scenario 4: Agile Data Analysis with Chat BI

On the consumption side, Data Agent offers two Chat BI modes. The quick query mode answers simple metric requests with minimal token usage and fast response. The deep analysis mode invokes stronger AI models to provide insights, action recommendations, and a complete data report for complex analyses such as attribution or trend studies.

Architecture Iterations

The Agent architecture has undergone three major changes in the past two‑three years:

Phase 1 – Dify‑based : focused on AI workflow orchestration; the Agent’s autonomy was limited but the ecosystem was rich.

Phase 2 – AgentScope : an open‑source Alibaba framework emphasizing multi‑Agent collaboration; highly flexible for exploratory development, though the ecosystem was smaller.

Phase 3 – Dual‑engine (QwenCode + OpenClaw) : QwenCode/Cloud Code deliver strong developer‑side code generation but require extensive engineering for enterprise readiness; OpenClaw adds multi‑channel IM integration and a “growth‑type” Agent that improves with usage, though its enterprise capabilities are still maturing.

DataWorks Data Agent 2.0: Cloud‑Native Fully Managed Practice

Developer‑side agents such as CloudCode and QwenCode typically run on personal machines and cannot provide 24/7 service, posing security, risk, and compliance challenges in enterprises. Data Agent 2.0 builds a cloud‑hosted sandbox based on QwenCode, enabling continuous operation, secure connectivity to production systems, and dedicated permission auditing.

The system offers four interaction modes:

Chat UI : standard natural‑language dialogue.

CLI : web‑terminal suited for developers and power users.

Remote Control : scan a QR code to open the same Agent interface on a mobile device, synchronizing the session like Apple’s Continuity.

IM Channel : integrates with DingTalk, Feishu, and WeChat Work via OpenClaw‑style channels.

AI Assistant Service: Secure, Controlled Operations Assistant

Built on OpenClaw, the AI assistant addresses key enterprise concerns:

Fully managed, no‑ops : one‑click instance launch, 24/7 availability.

Security and control : private connectivity to IM endpoints using Alibaba Cloud Global Acceleration, PrivateZone, and PrivateLink; all traffic stays off the public internet, and write actions require double confirmation.

Built‑in official Skills : cover data operations, governance, task diagnosis, workspace diagnosis, alert handling, quality monitoring, etc.

In practice, when a task fails and triggers an alert, the assistant pushes the alert to the IM channel. The user can diagnose, perform root‑cause analysis, and remediate directly from the chat without opening a PC. For example, the assistant can identify a failed task caused by an expired resource group and offer a one‑click retry after the resource is updated.

Case Study: Taobao Flash Sale

Within Alibaba, the Taobao Flash Sale business integrated with DataWorks Data Agent. The legacy workflow required hours to days of manual IDE‑based development, suffered from fragmented steps, inconsistent standards, limited quality checks, and poor knowledge reuse. After switching to the Data Agent mode, end‑to‑end intelligent development covered the whole pipeline. By extending custom Skills and a business knowledge base, development cycles shrank from 12‑23 hours to 5‑10 minutes. Standards were enforced automatically via Skills and Workflows, quality was ensured through systematic checks, and best‑practice knowledge was captured in reusable Skills.

The transformation is not merely a tool upgrade; it fundamentally changes how a data platform operates. When development time drops from days to minutes and governance becomes a self‑sustaining loop, the platform’s value chain shifts dramatically. Whether the Agent is ready for large‑scale rollout depends on an enterprise’s willingness to inject standards, knowledge, and best practices into the Agent’s evolution loop.

References

Official documentation: https://help.aliyun.com/zh/dataworks/user-guide/new-data-agent

Product entry: https://dataworks.data.aliyun.com/product/agent

Additional resources: https://www.aliyun.com/activity/bigdata/dataagent/skills