DeepEye: Building an Autonomous, Human‑Steerable Data Agent System

DeepEye is an open‑source autonomous data‑agent system developed by the HKUST (Guangzhou) research group, designed around two core principles: autonomy and harnessability. The platform aims to let agents automatically understand data, generate hypotheses, execute analyses, and produce insights while preserving human expert judgment at critical reasoning steps.

1. Introduction

Data agents are positioned as exploratory "scientists" that tackle open, complex data‑analysis problems, unlike coding agents that focus on well‑defined tasks. By integrating large language model (LLM) semantic reasoning with automation engines, DeepEye seeks to automate the full data‑analysis pipeline—from acquisition and cleaning to insight generation.

2. Hierarchical Taxonomy for Data Agents (L0–L5)

The authors adopt the SAE J3016 autonomous‑driving grading scheme to define six capability levels for data agents:

L0 : Fully manual analysis (traditional ML era).

L1 : Basic interactive query (e.g., ChatBI).

L2 : Automated data preparation and simple analysis, still fully supervised.

L3 (conditional autonomy): Agents autonomously compose pipelines (data lake, context‑aware planning, LLM‑driven task execution) but require human oversight for anomalies.

L4 (high autonomy): Agents proactively monitor data lakes, discover high‑value tasks, and generate insights with minimal human intervention.

L5 (full autonomy): Agents discover unknown patterns, formulate scientific hypotheses, and drive data‑centric breakthroughs without human involvement.

This taxonomy clarifies the transition from human‑centric to agent‑centric decision making and helps set realistic expectations for research and product roadmaps.

3. Data Agents vs. General LLM Agents

General LLM agents excel at content generation or task automation, whereas data agents focus on end‑to‑end data‑analysis workflows that must handle heterogeneous structured, semi‑structured, and unstructured sources. Key challenges include semantic ambiguity (e.g., field name "WZRL" meaning different things in gaming vs. pharma) and error cascade sensitivity, which demand tight integration with databases, stream‑processing engines (e.g., Flink), SQL validators, and visualization tools, all under a human‑in‑the‑loop safety net.

4. Vision for Proactive and Generative Data Agents (L4–L5)

At levels 4 and 5, agents become proactive: they continuously monitor data lakes, discover valuable tasks, orchestrate pipelines, and even invent new analytical methods when existing ones fail. The authors describe a three‑stage "discover‑innovate‑execute" loop that eliminates human intervention entirely, marking a shift from automation to generative intelligence.

5. System Architecture

DeepEye’s backend is built on a workflow‑engine that compiles user‑provided JSON plans into a directed acyclic graph (DAG) covering data ingestion, cleaning, modeling, and visualization. Core components include:

Knowledge Search : Retrieves relevant domain knowledge.

Code Executor : Runs generated SQL, scripts, or other tool commands.

Dashboard Generator and Report Generator : Produce interactive visualizations and structured reports.

The engine validates DAG topology (e.g., cycle detection), enforces security policies, and performs schema‑semantic matching before optimization identifies parallelizable nodes for efficient execution.

6. Human‑in‑the‑Loop and SOP Learning

Users can intervene on specific nodes (e.g., NL2SQL) to adjust workflows dynamically. DeepEye also learns standard operating procedures (SOPs) from high‑accuracy historical workflows, creating reusable templates that reduce development cost and improve explainability.

7. Knowledge Module and Memory

A knowledge module aggregates SOP experience, meta‑data definitions, and domain documents to align semantics across heterogeneous sources. Long‑term memory combines vector databases, metadata management, and knowledge graphs, enabling the agent to retain context across sessions and support continuous learning.

8. Implementation Highlights

DeepEye operates on a model‑agnostic, no‑base‑model training architecture, focusing on task orchestration to maximize LLM capabilities while avoiding costly retraining. The system supports dynamic knowledge ingestion (Markdown, JSON, databases, PDFs) and employs node‑level context compression and task decomposition to mitigate LLM context explosion.

Overall, DeepEye demonstrates a concrete pathway from manual data analysis toward fully autonomous, generative data agents, providing a reference architecture, capability taxonomy, and open‑source implementation (https://deepeye.hk, https://github.com/HKUSTDial/DeepEye) for the research community.