How OxygenData Built a Harness for AI-Native Data Warehouse Development

Introduction: From Vibe Coding to Harness

Over the past half year, AI coding assistants have been used mainly for code completion, unit tests, and bug fixes. In data warehouse development, the most direct application is generating SQL from requirements. While SQL generation saves time — turning half-day efforts into minutes — delivering a data requirement involves far more: clarifying PRDs, confirming metric definitions, assessing asset reuse, designing technical solutions, creating tables, trial runs, data validation, task publishing, scheduling verification, and communicating deliverables and risks. A mistake at any step renders fast SQL generation meaningless.

The team initially adopted Vibe Coding: feeding requirements to a model, iterating through chat, and producing SQL. However, three problems emerged: each new session started from scratch; models hallucinated plausible but incorrect metric definitions; and context had to be manually shuttled across tools. The goal shifted from building a better SQL assistant to embedding an Agent into the real data warehouse R&D environment — one that reads trusted context, follows fixed procedures, calls real platforms, pauses for human confirmation at risk points, and can resume from saved state and evidence after failures. This is the AI-native data warehouse R&D vision behind OxygenData's Harness.

Three Evolutionary Phases

Phase 1: Accelerating SQL Generation

The first approach was straightforward: paste requirements, add table schemas, let the model generate SQL, and have developers manually review and publish. This worked for single-point development and quickly demonstrated the model's code generation potential. However, quality depended heavily on the user. Experienced developers knew what context to provide and what to verify; novices tended to accept the first answer. The same task yielded widely different results depending on who performed it.

Phase 2: Skills and MCP for Single-Point Capabilities

Next, the team decomposed experience — requirement clarification, metadata querying, script development, trial runs — into Skills, and connected them to platform systems via MCP (Model Context Protocol). Agents could now query table schemas, lineage, scripts, and task statuses instead of relying on user-provided snippets. This solved "can it do it" and "can it touch the real environment," but not "who does what first." Each Skill completed its own step, yet cross-stage handoffs still required human judgment. When a step failed, there were no unified rules for continue, rollback, or redesign.

Phase 3: Harness for End-to-End Orchestration

The real change came when Orchestrator, TRD, Pipeline, manifest, and gates were combined. Requirements are clarified first, existing assets explored; TRD makes global technical decisions; physical tables, datasets, and dashboards are executed by their own Pipelines; each stage leaves artifacts, state, and platform evidence. Downstream consumes only confirmed upstream conclusions; if a design gap appears, it returns with evidence to TRD for revision instead of ad-hoc fixes during execution. AI shifts from a generation tool to a participant working under engineering constraints — the dividing line between Vibe Coding and Harness.

Harness Architecture: Four Layers

The architecture comprises four layers, plus two cross-cutting capabilities:

Knowledge & Context Layer : business definitions, model assets, metrics, lineage, development standards, historical decisions — the basis for Agent judgment.

Process Layer : Skills for PRD clarification, asset exploration, TRD, development, verification, delivery — defining how each step is done and what constitutes completion.

Orchestration & Execution Layer : Orchestrator, TRD, and separate Pipelines for physical tables, datasets, and dashboards — determining execution order and how plans are decomposed and run.

Platform Connectivity Layer : MCP capabilities for JoySpace, DongDP, DongMetrics, Buffalo, BI, etc. — enabling reading real facts and performing real operations.

Cross-cutting: manifest and stage artifacts preserve state, evidence, and resume points; verification, gates, and authorization define completion standards and constrain high-risk operations. These layers are not independent systems: knowledge aids understanding, Skills constrain process, MCP connects platforms, Orchestrator sequences, Pipelines deliver, manifest records, and verification decides whether the chain proceeds.

Orchestrator: Order Without Execution

The Orchestrator does not write SQL, create tables, or build dashboards. It orchestrates global order: identifying which stages to run, validating upstream artifacts, scheduling whitelisted Skills, pausing at key nodes for confirmation, recording stage status, and aggregating delivery evidence. The current end-to-end chain follows this phase contract:

0 Orchestration Init : establish artifact directory, execution plan, main manifest — ensures traceability and resumability.

1 PRD Clarification : converge business language into developable requirements — gate: requirement status confirmed.

2 Asset Reuse Exploration : identify intent across dashboards, datasets, physical tables, and ODS layer; explore current state — each asset gets an iterate, create, or no-action conclusion.

3 TRD Design : form unified technical solution for physical tables, datasets, dashboards — gate: explicit user "confirm TRD."

4 Physical Table Pipeline : develop scripts, create/alter tables, trial run, dual validation, publish tasks, schedule re-verification, optional DQC config — returns real execution results.

5 Dataset Pipeline : data preparation, preview, field processing, save and publish — returns real execution results.

6 Dashboard Pipeline : page building, preview, revision, acceptance, publish — returns real execution results.

7 Delivery Summary & Knowledge Reflow : aggregate delivery evidence, incomplete items, reusable knowledge — produces delivery report and knowledge reflow receipt.

The chain does not run every stage rigidly; the real path is decided by asset intent and TRD.

Skills as Process Contracts

A Skill is not a long prompt teaching the model how to think, but a process contract specifying trigger conditions, required inputs, output artifacts, forbidden actions, and completion criteria. For example, the clarification Skill only converges business descriptions into computable requirements; the asset exploration Skill returns candidates and evidence but does not make business trade-offs; the physical table design Skill forms a plan but does not create tables; the trial run Skill only runs existing tasks, not create them. Clear boundaries make Skills composable and debuggable; a "super Skill" that does everything becomes an unmaintainable giant prompt.

MCP: Grounding Decisions in Real Platforms

MCP wraps platforms (JoySpace, DongDP, DongMetrics, Buffalo, BI) into standard tool interfaces. Skills express "what to query" and "what to accomplish"; authentication, parameters, and platform calls are handled by MCP. Beyond eliminating copy-paste, the key value is that Agents read real-time metadata within authorized scope and judge state by platform return values. Without a real task ID, the task is not created; while a trial run is still running, success cannot be declared prematurely.

TRD Design, Pipeline Execution

Physical tables, datasets, and dashboards are executed by three separate Pipelines. Design and execution layers hand off not via chat history but through TRD's fixed chapters and a unified pipeline_result structure.

Physical Table Pipeline consumes TRD's model solution, handles scripts, schema, trial runs, dual validation, task publishing and trial runs, conditional DQC, and production handoff.

Dataset Pipeline consumes dataset solution, handles data preparation, preview, field processing, save and publish.

Dashboard Pipeline consumes dashboard solution, handles page building, preview revision, acceptance, and publish.

If execution reveals a TRD gap, the Pipeline does not patch a definition and continue; it returns blocked(trd_gap) with evidence back to TRD for revision. This ensures clarity on who made the decision, which version the user confirmed, and what standard governs acceptance.

TRD as the Architectural Brain

Requirement clarification answers "what the business wants"; asset exploration answers "what exists." Neither directly guides execution. A data-architect-level decision layer is needed to answer holistically: how the entire data pipeline should change, why, how layers connect, where risks lie, and how to verify. In OxygenData, TRD plays this role.

An earlier attempt let physical table, dataset, and dashboard Agents each design independently. Locally each made sense, but combined they produced inconsistencies: physical table at order granularity, dataset at user-day granularity; upstream partitioned by payment time, dashboard filtered by order time; table updated T+1 but dashboard titled "real-time." The issue was not executor capability but lack of global architectural trade-offs. A data-architect TRD must answer six question categories:

Requirement Abstraction : which delivery assets correspond to the business ask, which are merely mentioned or affected.

Current State Diagnosis : to what extent existing tables, metrics, datasets, dashboards can be reused, and where gaps lie.

Overall Design : data sources, processing layers, granularity, and timeliness to consumers.

Cross-Layer Constraints : consistency of primary keys, granularity, definitions, refresh cycles across physical tables, datasets, metrics, dashboards.

Risk Trade-offs : why reuse, extend, or rebuild; cost, performance, SLA, impact scope trade-offs.

Execution & Acceptance : how to split the plan across Pipelines, what evidence each layer needs for completion.

Fixed chapters, asset evidence, cross-layer constraints, and confirmation gates have turned TRD from a descriptive document into the sole plan input for downstream execution. Next steps: strengthen cross-artifact consistency checks to catch granularity, field, time-definition, and refresh-cycle conflicts early; accumulate real architectural decision cases so TRD explains why a design fits, not just fills a template. SQL, DDL, and deployment can be increasingly tooled, but only sound TRD architectural judgment gives meaning to automated execution.

Five Stabilizing Design Principles

5.1 Asset Reuse Before Creation

AI excels at generating new artifacts and tends to "see a requirement, create new." In a data warehouse this means duplicate tables, metrics, datasets. Therefore, asset exploration first judges whether dashboards, datasets, physical tables, and ODS layers have change intent. Only those with intent are explored further; others are explicitly skipped. The system surfaces candidate assets, similarity evidence, and upstream/downstream impact; final reuse decisions are made by TRD technical evaluation. This apparent detour avoids costlier rework and governance later.

5.2 Human Confirmation Only at Critical Decisions

Not every step needs human input. Excessive confirmations turn the Agent into a popup-heavy process bot, drowning real risks in a sea of "continue?" prompts. Human confirmation is concentrated on two categories: business trade-offs the model cannot reliably judge (e.g., which of several similar assets to reuse), and write operations with external impact (production table DDL, production partition writes, production task publishing, persistent DQC rules). Routine stage transitions, DEV environment writes, read-only validations, and result handovers proceed automatically. Production writes require per-item authorization showing object, environment, change, impact, and stop mechanism; a blanket "continue" cannot authorize all subsequent online actions.

5.3 Downstream Cannot Mutate Upstream Decisions

Agents easily make local optimizations during execution: tweaking SQL because TRD feels awkward, swapping a metric definition while building a dashboard. Locally clever, but engineering chaos ensues: who changed the plan, which version did the user confirm, what standard governs acceptance — all become untraceable. Hence the principle of unidirectional artifact flow: downstream consumes only confirmed upstream artifacts; problems are returned with evidence, not by rewriting upstream conclusions.

5.4 Manifest as Source of Truth, Not Chat

A data requirement may span hours or days, switch sessions or owners. Chat logs suit communication, not engineering state. Two-level manifests preserve the scene: the main manifest records overall chain stages, confirmation events, and summaries of the three Pipelines; each Pipeline's sub-manifest records internal state, platform IDs, log evidence, and resume points. After interruption, the Agent reads the manifest, validates upstream artifacts and versions, and resumes from a legal stage — no need to "recall where we left off." Team-shared facts must reside in artifacts and platforms, not in a conversation fragment.

5.5 Knowledge Loop: Turning Delivery into Context

What deserves retention from a requirement is not a trial-run log but stable business definitions, transformation logic, schemas, validation rules, and asset lineages. At chain close, a knowledge reflow Skill generates candidate knowledge cards from delivery evidence for user selection before submission. Next time a similar requirement arrives, the Agent starts from this knowledge instead of re-asking and re-guessing. The evolving knowledge initialization framework organizes multi-source metadata, lineage, and business semantics together, though continuous optimization is needed; capturing individual developers' tacit experience remains challenging.

The goal is to move knowledge from "searchable and reflowable" to "verifiable and governable." Every knowledge item should retain source, applicability scope, citation records, and verification evidence, progressing through draft, verified, production-proven states. Expired or conflicting knowledge must not silently influence plans. Data warehouse knowledge is inherently a relationship graph: which metrics a dashboard uses, which fields those metrics depend on, which tables provide those fields, which tasks produce those tables, why a TRD chose a certain pipeline, and by what rules it was accepted. Connecting these relationships lets the Agent's "reuse this table" recommendation return not just a similar name but a traceable decision chain.

Example: Shop Operations Dashboard

For a "shop operations dashboard" requiring exposure, click, visit, and conversion performance, the Agent first translates business terms into statistical definitions needing confirmation; then queries existing metrics, tables, datasets, dashboards, presenting reuse evidence. TRD unifies granularity, time definitions, processing chain, refresh cycle, and acceptance rules; after user confirmation, it dispatches to respective Pipelines. Humans handle business trade-offs and high-risk operations; Agents handle high-frequency information gathering, execution, and validation; platform return values determine real state. What was scattered across human minds and multiple systems is now threaded by Harness.

Knowledge as the Starting Point for Next Delivery

Models, Skills, and tools will update, but what remains reusable long-term are business definitions, architectural decisions, data models, validation rules, and failure experiences verified through repeated delivery. These are not "forever correct" — they have applicability scopes and expire as business and systems evolve. The knowledge loop's two ends are connected: at chain start, fetch minimal necessary context for the current stage; at chain end, generate knowledge candidates from delivery evidence for user selection and reflow. PRD stage needs business concepts and definitions; TRD stage needs architectural decisions and asset relations; development stage needs schemas, standards, and risk alerts. There is no need to stuff the entire knowledge base into the Agent at once.

Unsolved Challenges

8.1 From Single Scripts to Workflow Jobs

Current delivery targets mainly single-script standard tasks. For multi-node workflows with dependencies and branching, the chain typically delivers only to script level; subsequent node orchestration, dependency configuration, and overall publishing remain manual. This is not merely calling task creation APIs multiple times. Workflows need new plan representation and execution contracts: TRD must describe node topology, upstream/downstream dependencies, runtime parameters, failure strategies; Pipelines must support full-graph validation, partial reruns, pre-publish checks; manifests must record node-level state. Only with these can Harness move from "deliver a script" to "deliver a production workflow chain."

8.2 Reducing Dialogue Rounds, Improving First-Pass Quality

Step-by-step clarification reduces misjudgment but asking every question equally drags the chain. Users who find the process verbose revert to Vibe Coding. The direction: keep separate confirmation for questions that change technical approach or have dependencies; merge independent, lower-risk questions into a structured checklist. After each stage, compress confirmed facts into artifacts and manifest; downstream receives only the minimal context it currently needs, not the entire chat history again. Meanwhile, requirement understanding, TRD, and SQL still often need multiple revisions. Next steps: add cross-artifact consistency checks and real-case evaluation sets; continuously monitor first-pass rate, average revision rounds, and frequent errors. Stability cannot rely on "please check carefully"; it must be built through structure, validation, evaluation, and feedback loops.

8.3 Unified Safety Control for Production Writes

Individual execution Skills already have confirmation gates for production table DDL, partition writes, task publishing, and DQC persistence, but controls are scattered. As writable platforms and operation types grow, per-Skill rule maintenance risks inconsistent definitions or over-broad authorization. A unified online safety control plane is needed: default read-only and DEV-only; production operations granted minimal authorization by object, action, environment; pre-execution impact display, post-execution audit events; re-confirmation on parameter or object change; support for pause, timeout, and human takeover for high-risk ops. Confirmation becomes a scoped, traceable authorization, not a generic "continue?"

Conclusion

Vibe Coding revealed the model's generation efficiency in data warehouse development and helped quickly identify the most automatable local tasks. But when the goal shifts from "write a SQL snippet" to "deliver a data requirement," the problems extend beyond code generation: where does context come from, who owns the solution, how do stages hand off, where to resume after failure, what evidence constitutes completion, how are online operations controlled. OxygenData's Harness incrementally encodes these formerly experience- and memory-dependent concerns into a shareable engineering system for the team.

Skills constrain each step, MCP connects to real platforms, TRD owns global architectural decisions, Orchestrator maintains chain order, Pipelines deliver concrete assets, manifest preserves state and evidence, gates and human confirmations guard quality and safety. The solution is still evolving. Workflow jobs, artifact stability, knowledge governance, and unified online safety control all need further refinement. Ultimately, as tools grow stronger, human judgment must grow stronger too.

Reference: OxygenData Platform