Continuous Delivery for High-Quality Datasets: Dataset CI & Eval CI Pipelines

The previous article discussed how to attribute AI errors to data, semantics, retrieval, model, prompt, or tool execution. Once fixes are made, a practical question remains: how do modified data, semantics, rules, prompts, and indexes re-enter production? Many teams simply rerun the data job and consider a green scheduler, populated target tables, and rebuilt vector indexes as proof of releasability. However, pipeline success only proves execution completion; it cannot guarantee the new data still satisfies contracts, that old policies were not incorrectly overwritten, that RAG citations have not degraded, or that agents will not trigger unauthorized actions due to new knowledge.

High-quality dataset continuous delivery is not about running ETL more frequently, but connecting data validation, task evaluation, security gates, human approval, version release, and rollback evidence into a repeatable engineering pipeline.

This article uses a weekly policy knowledge base update as a running case to explain how Dataset CI and Eval CI collaborate. Dataset Contract, Manifest, and Registry details were covered in article 04; platform architecture and component responsibilities will be discussed in article 10.

ETL Success Does Not Mean Data Product Is Releasable

Assume an enterprise syncs the latest policy documents to a knowledge base weekly. This week a new "Travel Reimbursement Policy (2026 Revision)" was uploaded, parsing succeeded, chunk counts looked normal, and the vector index rebuilt successfully. From the data job perspective, the update is green. Yet it may still have the following problems:

The new policy is not yet effective but overrides the still-valid old policy in current Q&A.

Parsing lost critical negation words such as "must not" or "except".

Document source and approval status are missing from metadata, so there is no proof it is an official version.

New and old policies are split into similar chunks; retrieval recalls both, and the model mixes two periods' wording.

Answers remain fluent but citations point to expired clauses.

Agents generate current reimbursement advice based on future policies and even attempt to call business tools.

None of these issues cause the ETL job to fail. Table existence, reasonable row counts, and successful indexing only prove technical steps completed, not that the data product possesses correctness, independence, and business usability.

Continuous delivery must simultaneously answer three questions:

Is the candidate data version itself qualified? : Do structure, semantics, source, distribution, isolation, and security satisfy the contract?

Do the AI capabilities that depend on it still hold? : Have model, RAG, and agent task effects passed regression?

Is the release process controllable? : Who approves, which versions are released, how is canary performed, and can the whole system roll back on failure?

Continuous Delivery Is Not a Single Pipeline but Three Controlled Chains

A runnable high-quality dataset delivery process contains at least three interconnected but differently responsible chains:

Candidate Build : Inputs are source files, database snapshots, semantics, and processing code. Tasks: extraction, parsing, standardization, chunking, annotation, index building. Output: immutable candidate version. Cannot prove usability for AI.

Dataset CI : Inputs: candidate version, Contract, quality and security policies. Tasks: schema, constraint, semantic, distribution, leakage, and permission checks. Output: data validation report. Cannot alone prove task effectiveness.

Eval CI & Release Control : Inputs: candidate version that passed data checks, evaluation baseline, dependency versions. Tasks: model/RAG/agent regression, approval, canary, and rollback. Output: releasable baseline and evidence package. Cannot replace online business validation.

After candidate build succeeds, it should not directly overwrite production tables or indexes but generate a candidate version with a unique version number and content fingerprint. All subsequent tests, approvals, and evaluations target this immutable candidate. Only after Dataset CI passes can the candidate enter task evaluation; only after Eval CI and required approvals pass does it become a releasable baseline. Real business metrics after launch must continue to be observed; offline pipelines cannot substitute for them.

Dataset CI: First Prove Candidate Data Does Not Break Existing Contracts

Dataset CI is not merely running null checks on a few tables; it verifies whether candidate data still satisfies its intended purpose. For a policy knowledge base, at least six categories of checks are needed.

1. Schema and Structure Checks

Verify fields, types, enums, primary keys, time formats, and nested structures are compatible. For example, each policy document must have a unique document ID, each chunk must trace back to original page numbers or clause positions, and effective dates must not be parsed as free text. Structure checks answer whether data looks as agreed; they are the minimum bar and do not guarantee business meaning correctness.

2. Source and Lineage Checks

Every policy must trace back to formal source, approval records, original file fingerprint, and parsing version. Missing source, missing approval status, or unexplained changes to the original file should block release. For a knowledge base, "found a similar file on the web" does not make it valid enterprise policy evidence. Source authority is part of data quality, not a post-release manual note.

3. Semantic and Temporal Constraint Checks

Check policy type, applicable organization, applicable role, effective time, expiration time, and version replacement relationships. Future-effective policies can enter the candidate library early but must not replace current policies at the current time point. Policy Q&A must distinguish "what is asked now" from "what business occurrence time is asked about". For example, on August 25 asking current reimbursement rules should cite the current version; querying a business event on September 2 may need the version effective September 1.

4. Content Integrity and Parsing Quality Checks

Verify page counts, paragraphs, tables, clause numbers, attachments, and key symbols are complete, with focused validation on negation words, currency units, exception conditions, and cross-page tables. Parsed text that "looks like it has content" is not sufficient. If the original "not exceed" loses the "not", the system acquires not minor noise but a business rule with opposite direction.

5. Distribution, Duplication, and Anomaly Checks

Compare candidate version with baseline on file count, chunk count, length distribution, topic coverage, organization coverage, and duplicate clusters. Distribution changes are not necessarily errors but must be explainable. If only one policy is added yet tens of thousands of chunks appear, the chunker may be abnormal; if new files highly duplicate historical policies, confirm whether it is a formal replacement, partial revision, or duplicate ingestion.

6. Isolation, Security, and Leakage Checks

Check sensitivity levels, access scopes, cross-domain boundaries, and evaluation isolation. Evaluation questions and their standard answers must not enter the knowledge index under evaluation; restricted policies must not expand visible scope due to index rebuild. Permission checks cannot stop at file-level ACLs; retrieved chunks, caches, cited content, and agent contexts must inherit corresponding access constraints.

Eval CI: After Data Passes, Prove Task Capability Has Not Regressed

Dataset CI passing only means candidate data satisfies the data contract. Whether it is truly usable requires evaluation on the corresponding tasks. Training, RAG, and agents consume data differently; they cannot share a single "overall accuracy" for acceptance.

The following table maps consumption modes to Eval CI focus, policy knowledge base examples, and key boundaries:

Model Training or Classification : Focus on label stability, stratified metrics, challenge sets, key category regression. Example: policy type and applicability classification regression. Key boundary: must use independent evaluation set.

RAG : Focus on recall, ranking, evidence coverage, citation consistency, temporal correctness. Example: current and future policies correctly recalled by business time. Key boundary: do not substitute generation fluency for evidence correctness.

Agent : Focus on judgment, permissions, tool selection, action contracts, and side-effect control. Example: only generate controlled advice, do not erroneously call reimbursement tools. Key boundary: evaluation environment must not produce real side effects.

For this case, Eval CI should at least include the following regressions:

Current Policy Regression : Questions at current date still cite current policy.

Future Policy Regression : When a future business date is explicitly specified, the version effective at that time can be cited.

Expired Policy Regression : After a policy expires, it no longer serves as primary basis for current conclusions but remains available for historical traceability.

Negation and Exception Regression : Questions containing "must not", "except", "after approval" must not be interpreted inversely.

Citation Regression : Policy names, clauses, and original positions in answers are consistent.

Permission Regression : Different roles can only retrieve and cite content they are authorized to access.

Agent Safety Regression : Knowledge changes only affect suggestions and candidate judgments, never bypass approval to produce real business side effects.

On evaluation failure, retain failed samples, candidate evidence, model outputs, and dependency versions, and perform attribution using the method from "AI Errors: What to Fix". Do not automatically roll back data just because QA degrades, nor default the problem to the prompt team.

Which Issues Auto-Block, Which Require Manual Approval

Automation aims not to eliminate human judgment but to automatically enforce deterministic boundaries and hand changes needing business interpretation to clearly assigned owners.

The following situations typically serve as mandatory blocking items:

Contract or schema exhibits undeclared incompatible changes.

Formal source, approval status, file fingerprint, or lineage missing.

Effective/expiration conflicts cause multiple unexplained valid versions for the same organization at the same time point.

Critical clauses, negation words, currency units, or attachments fail to parse.

Leakage occurs between training and evaluation, production and test, or different permission domains.

High-risk evaluation metrics fall below business red lines.

Dependency versions for data, semantics, index, prompt, model, or rules are incomplete.

The following situations usually require human approval rather than mechanical blocking:

Policy revisions cause explainable changes in topic, length, or chunk distribution.

New policies require simultaneous changes to standard answers, business rules, or agent advice boundaries.

Low-risk metrics fluctuate slightly but high-risk capabilities and safety thresholds remain satisfied.

New policies partially conflict with historical policies; business owner must confirm applicability scope.

Few new issues appear during canary; need to judge whether they are expected changes or system regressions.

Thresholds cannot be decided ad hoc by the pipeline team. Each check must bind a responsible party, threshold source, failure action, and exception approver. High-risk red lines must not be bypassed via ordinary approval; if an exception is truly needed, record time limit, compensating controls, and exit conditions.

A simplified quality gate configuration can be organized as follows:

dataset_release_gate:  dataset_id: policy-knowledge-base  candidate_version: 2026.08.25-rc1  baseline_version: 2026.08.18  policy_ref: policy-kb-release-gate-2026Q3  blocking_checks:  - schema_compatibility  - source_and_approval_traceability  - effective_time_consistency  - critical_content_integrity  - evaluation_isolation  - permission_boundary  task_evaluations:  - current_policy_rag_regression  - future_effective_policy_regression  - citation_consistency  - agent_no_side_effect  manual_approvals:  - business_policy_owner  - dataset_product_owner  - security_owner  on_failure:    action: block_release    route_to: release-incident-queue

The example only shows configuration structure, not fixed check items or universal thresholds. Specific thresholds must be jointly determined by scenario risk, historical baselines, and business tolerance.

Release Must Lock a Set of Versions, Not Just Dataset Version

The policy knowledge base result depends not only on documents but also on parser, chunker, semantic model, embedding model, index, retrieval config, prompt, generation model, rules, and tool contracts. Article 04 explained that a single version number cannot represent all release dependencies. This article further emphasizes: the continuous delivery pipeline must read and verify the full Manifest, ensuring the evaluation object and the final release object are the same set of engineering artifacts.

release_baseline:  release_id: policy-kb-release-20260825-01  dataset_version: 2026.08.25  contract_version: 3.1.0  semantic_model_version: policy-domain-2.4.0  parser_version: policy-parser-1.8.2  chunking_profile: policy-chunk-v5  embedding_model_version: embedding-release-2026-07  index_version: policy-index-20260825  retriever_config_version: retriever-policy-4.2  prompt_version: policy-assistant-3.6  generation_model_version: model-release-2026-08  rule_version: policy-effective-time-2.1  tool_contract_version: reimbursement-advice-1.4  evaluation_baseline: policy-eval-2026Q3

Not every data update must modify all dependencies, but every release must explicitly declare which versions it actually uses. Otherwise, offline evaluation passes one combination while production runs another.

Canary and Rollback Must Execute by Complete Baseline

After a candidate passes gates, it should not immediately replace all production consumers. The policy knowledge base can first enter a shadow environment, using de-identified copies of real queries for side-by-side comparison; then canary by organization, role, or question type, not just random traffic splitting.

A controlled release path can be divided into:

Candidate Isolation : Candidate files, structured data, and indexes separated from production versions.

Offline Evaluation : Complete Dataset CI and Eval CI on the locked evaluation baseline.

Shadow Validation : Receive production query copies but do not return results to users or execute real actions.

Small-Scale Canary : Open to specific organizations, roles, or low-risk question types.

Metric Observation : Monitor citation errors, refusals, human corrections, permission blocks, and tool failures.

Full Release or Rollback : Promote when observation conditions met; trigger rollback to previous complete baseline when red lines hit.

Rollback cannot merely switch the vector index alias back to the old version. If the new version also modified semantic mapping, prompt, or rules, rolling back only the index creates a never-evaluated combination. Therefore, the rollback object should be the complete release baseline. After recovery, replay the queries and actions that triggered the rollback to confirm the previous version indeed restores expected capabilities, not just that the service becomes available again.

Release Evidence Is Not Log Attachments but Part of Delivery Result

After continuous delivery completes, an auditable release evidence package should be formed, including at least:

Change Description : What sources and business calibers were added, modified, or expired.

Candidate Version Manifest : Content fingerprints, processing code, and full dependency Manifest.

Dataset CI Report : Structure, source, semantic, distribution, isolation, and security check results.

Eval CI Report : Model, RAG, agent baseline comparisons, failed samples, and risk conclusions.

Approval Records : Responsible persons, approval basis, exception items, validity periods, and compensating measures.

Release and Canary Records : Environments, scopes, times, observed metrics, and promotion conclusions.

Rollback and Replay Evidence : Rollback baseline, trigger cause, recovery results, and regression samples.

These artifacts serve both audit and baseline comparison for the next change. A "release success" without an evidence package only proves the system currently runs, not why it was allowed into production at that time.

Not All Datasets Need Heavy Pipelines

The rigor of continuous delivery should match change frequency, usage scope, and business risk. For one-off analyses, low-risk internal experiments, or small static knowledge sets, versioned files, automated basic checks, and manual review suffice; there is no need to build a complex platform first.

Full Dataset CI and Eval CI are more worthwhile when the dataset exhibits:

Frequent updates, with multiple models, RAGs, or agents consuming it jointly.

Errors affect customer rights, financial processing, safety production, or compliance judgments.

Data, semantics, prompts, models, rules, and indexes often change in tandem.

Cross-organization release, canary, rollback, and audit are required.

Online feedback continuously feeds into training, evaluation, or knowledge base updates.

Even without a full platform, versioning, ownership, release checks, and rollback plans cannot be omitted. The tooling is lightweight; the governance responsibility is not.

What a Runnable Continuous Delivery Outcome Should Contain

The final delivery cannot be just a scheduled job; it must include at least:

Dataset CI Process and Test Suites : Candidate build, structure, source, semantic, distribution, isolation, and security checks.

Eval CI Process and Evaluation Baselines : Model, RAG, agent task regressions and high-risk red lines.

Quality Gate Configuration : Automatic blocking items, manual approval items, threshold sources, and failure routing.

Complete Release Manifest : Versions of data, code, semantics, index, prompt, model, rules, and tool contracts.

Canary and Rollback Strategy : Candidate isolation, shadow validation, release scope, observation conditions, and complete baseline recovery.

Release Evidence Package : Test, evaluation, approval, deployment, monitoring, rollback, and replay records.

These artifacts turn dataset updates from a "run a job" event into a repeatable, verifiable, and accountable engineering capability.

Summary

The core of high-quality dataset continuous delivery is not scheduling ETL more frequently, but ensuring every change passes the same explainable gate.

Candidate data first passes schema, source, semantic, distribution, isolation, and security checks, then enters model, RAG, and agent task evaluations. Deterministic errors are auto-blocked; changes needing business interpretation go to human approval. Release locks the full dependency baseline, undergoes shadow validation and canary before promotion, and on failure rolls back the entire baseline while preserving replay evidence.

A truly mature data pipeline not only delivers data downstream but can prove why this batch of data can be used downstream, and when judgment is wrong, can restore the entire capability stack to the last trusted baseline.

Next article will discuss how these Contract, Registry, test, evaluation, approval, and feedback capabilities should collaborate with existing enterprise lakehouse, metadata, vector stores, model platforms, and agent platforms, rather than building another isolated system.