Datus-agent: AI‑Native Data Engineering Agent for Context‑Aware Analytics

Core Positioning and Design Philosophy

Context‑driven collaboration – automatically aggregates metadata, metric definitions, SQL history, and business documents into a continuously updated semantic map, providing a unified context for engineers and analysts.

Agent‑based delivery – encapsulates data capabilities as domain‑specific sub‑agents rather than static tables; each sub‑agent contains its own context, toolset, and access rules.

Full‑lifecycle intelligence – covers SQL development, data‑quality validation, metric management, and task deployment, with AI‑assisted SQL generation, model optimization, and real‑time monitoring.

Technical Architecture and Core Components

Interaction Layer

Provides three role‑specific entry points:

Datus‑CLI – AI‑native command‑line tool for engineers. Example: datus-cli --namespace demo Datus‑Chat – web‑based multi‑turn chat for analysts, includes issue reporting, success tagging, and an !export command to generate reports.

Datus‑API – stable REST interface for developers to embed data services in internal BI platforms or automation pipelines.

Context Engine Layer

Automatic context capture – continuously stores and indexes SQL history, table schemas, metric definitions, and business documents; supports both tree‑based and vector‑based retrieval. For example, a previously written “user monthly retention” SQL is automatically added to the context for reuse.

Sub‑agent system – creates domain‑specific sub‑agents. Example command: .subagent add mychatbot After creation, the sub‑agent can be configured with business scope, tool permissions, and access rules.

Continuous learning loop – uses analyst feedback and usage history to refine models and context. High‑scoring query results reinforce generation logic; flagged errors trigger automatic context adjustments.

Integration Layer

Supports data warehouses and databases such as StarRocks, Snowflake, BigQuery, Redshift, and Databricks SQL.

Integrates metadata and semantic layers including DataHub, Amundsen, Unity Catalog, AWS Glue, MetricFlow, dbt Semantic Layer, and LookML.

Connects with task schedulers and BI tools like Airflow, Dagster, Prefect, Looker, Tableau, Power BI, and Superset for direct synchronization of analysis results or automated pipelines.

Core Functions and Value

Context‑Aware Data Engineering

Automatically builds a semantic map that merges metadata, SQL comments, modification records, and business knowledge. The command /gen_semantic_model xxx creates a semantic model, which can be refined with @subject to combine AI‑drafted and manually corrected definitions. When analyzing user repurchase rates, the system links user and order tables, retrieves the historical SQL, and returns results without rewriting queries.

Domain‑Specific Sub‑Agents

Each sub‑agent encapsulates business context (e.g., an “e‑commerce” sub‑agent includes orders, users, and products), toolsets (SQL generation, charting, anomaly detection), and access rules, allowing analysts to query via the web UI without knowledge of underlying schemas.

End‑to‑End Automation

SQL Development – natural‑language to SQL conversion, automatic join optimization, and version control. Example: a request for “last month’s active users per province” yields a syntactically correct SQL with annotated tables and fields.

Data Quality – built‑in validation rules detect missing values, format errors, and logical contradictions, generating quality reports and supporting configurable alert thresholds.

Task Deployment – transforms SQL queries or analysis flows into scheduled jobs, integrates with Airflow, Dagster, etc., and supports retries and result notifications.

Lightweight Deployment and Efficient Collaboration

Implemented in Python 3.12; installation requires two commands:

pip install datus-agent==0.2.1

datus-agent init

The platform enables human‑AI collaboration: AI‑generated SQL can be edited manually, and the revised logic is fed back into the context to improve future generations. Analysts can rate results, report issues, or add comments, which the system uses to continuously refine sub‑agent behavior.

GitHub repository: https://github.com/Datus-ai/Datus-agent