Building a One-Person Data Team: Core Skills of a Full‑Stack Data Engineer

Why One‑person Data Team?

During a Data Engineering Open Forum in Silicon Valley, the author observed that despite AI increasing efficiency, managers expect more output, prompting a discussion on how an individual can maintain value in the “agent” era.

Core abilities of a full‑stack data engineer

According to Paul Ellwood, Head of Data Engineering at OpenAI, the most important skill is semantic ownership & responsibility —the power and duty to define business metrics, not just maintain them. This involves translating vague business language into clear data definitions, such as what constitutes an active user, GMV, or store revenue.

1. Semantic definition ability

The engineer must sit with stakeholders, clarify business logic, and formalize it into precise data models, because these semantic decisions dictate the organization’s data language.

2. Building an agentic data stack

Future data stacks are not just tool collections but orchestrations of continuously operating agents. Selecting appropriate lake‑warehouse architectures, streaming‑batch pipelines, quality frameworks, and schedulers, and integrating them into an agent infra is essential.

3. Constructing agent‑native context from history

Valuable data knowledge resides in past SQL, dashboards, documentation, and team discussions. Extracting lineage, hidden rules, and reliability indicators from this history creates the context needed for agents to operate correctly.

ChatBI and the “last mile”

Chat‑based BI tools often lack sufficient data‑engineering context, leading to inaccurate answers. Accurate results still depend on well‑defined tables, metrics, and reference SQL/templates. A practical approach is to augment existing dashboards with sub‑agents that can answer follow‑up questions while preserving context.

Validation loops for hands‑off automation

The real bottleneck is not SQL generation but validating that tables, metrics, and dashboards meet business requirements. Each company’s unique conventions mean that combining model capabilities with SQL review, data‑quality checks, and lineage knowledge is necessary to automate these “dirty work” tasks.

Data engineering harness: Datus 0.3

Datus is an open‑source data‑engineering agent designed to help a single engineer build end‑to‑end pipelines, define metrics, and deliver APIs, chatbots, or semantic layers. Version 0.3 adds:

Sub‑agents covering table, semantic model, metric, SQL, job, report, and dashboard generation, each with customizable validation specs.

Service adapters for Airflow scheduler and Superset/Graphana BI tools.

Support for various LLM providers (Codex OAuth, Claude, OpenRouter, MiniMax, GLM).

Streaming API for Datus‑Chat, lightweight web embed, Slack and Feishu channels.

Feedback‑driven memory for sub‑agents, enabling long‑term learning.

Reference templates to stabilize repetitive pivot queries.

Permission modes (normal/auto/dangerous) with fine‑grained tool access.

A proof‑of‑concept called AgenticDataTown demonstrates a full pipeline from BigQuery to Iceberg, then processing with DuckDB, StarRocks, Airflow, and Superset, illustrating how a human can focus on task specification while the agent handles execution and review.

Getting started

Readers can try the Datus‑studio playground (8 demo scenarios) and explore the GitHub repository, quick‑start guide, and e2e pipeline documentation. Additional resources include a VSCode plugin (planned) and enterprise collaboration options.