Agentic Data Nirvana: Rebuilding Search/Recommendation Data Production with AI Agents

01 Why Build This System

Current business scenarios lack a unified, complete data development platform. Data ingestion, cleansing, business processing, index building, and operations remain highly manual. The main site has no dedicated platform for the full data R&D flow; the recommendation system has a relatively mature index building system but still requires manual work for ingestion, cleansing, and business processing before ingestion; vertical sites rely on configuration control systems but have internal process differences requiring per-business manual development. These problems cause duplicate construction, high labor costs, and difficulties in unifying R&D standards, process handoffs, quality assurance, and operations. The goal is to build a unified data R&D system that connects ingestion, cleansing, business processing, index building, task submission, scheduling, and fault handling, preserving flexibility for different business scenarios while precipitating reusable platform capabilities.

02 System Panorama Design

The architecture is not merely adding a chat entry to the old pipeline, but making the intelligent entry, domain semantics, unified foundation, and quality assurance each occupy their proper place and collaborate as a system.

Horizontal main pipeline: Forge Agent organizes the entire R&D process; the domain data R&D platform precipitates business semantics through two main lines — feature engineering (feature definition, computation selection, publishing) and base pool processing (domain DSL, semantic compilation & planning, task submission & management). ADF serves as the unified data and compute foundation, carrying operator systems, connections & adapters, data & metadata, and multi-engine execution. Artifacts ultimately serve search, recommendation, key scenarios, and online applications. Natural language intent flows through domain semantics, execution DAG, data artifacts, and back to business feedback.

Vertical quality penetration: The data quality system does not belong to a single layer but vertically penetrates the full chain — data reconciliation, data quality, lineage tracking, and monitoring dashboards respectively provide consistency, compliance, traceability, and observability to underpin trust for the whole system.

03 Forge Agent Organizes the Data R&D Process

This solution does not simply add a conversational entry to the existing platform; it reorganizes the scattered capabilities in data R&D. Query, verification, planning, preview, submission, and status check are first encapsulated as tools with explicit parameters and stable returns; data sources, transformation relationships, and delivery targets are uniformly expressed via domain DSL; Forge Agent then invokes these capabilities according to user needs, stringing together the complete R&D flow.

Layer responsibilities:

R&D Entry (Forge Agent): Requirement clarification, task decomposition, Skill routing, tool selection, result interpretation & human handoff

Tool Interface (Sindri Plugin/MCP): Encapsulate context query, DSL construction, verification, planning, submission, and status query as stable tools

Domain Backend (Sindri Backend): Parse domain semantics, bind platform objects, generate execution plans, build requests and submit

Execution Foundation (ADF): Create DAG, organize node dependencies, schedule & run, maintain task & instance state

To make this flow stable for production, Forge Agent implements the following mechanisms:

Skill: Precipitates operation flows and domain knowledge for specific scenarios.

Plugin/MCP: Executes standard operations such as compilation, verification, generation, submission, and query.

Hook: Executes permission verification, audit, and human confirmation at critical nodes like submission.

Trace/Trajectory: Records actual invocation processes for problem localization and retrospectives.

Golden Case/Eval: Verifies via fixed cases whether version changes affect existing capabilities.

Through this division of labor, the model only handles requirement understanding and flow organization, while production operations remain constrained by platform interfaces, permission rules, and execution results. This reduces manual cross-system operations while preserving the determinism of existing platforms in planning, scheduling, permissions, and auditing.

04 Sindri: From Domain Semantics to Trusted Execution Plan

Forge Agent faces humans to organize the R&D process; Sindri faces the platform to interpret "how exactly this data processing should be executed." The two interact via deterministic tools, not by letting the model directly assemble ADF requests.

A typical link can be summarized as:

Requirement clarification → Context acquisition → DSL construction → Structural verification → Sindri authoritative planning → Submission preview → Explicit confirmation → Sindri submission → ADF lifecycle → Status feedback.

4.1 DSL Expresses "What to Do"

Domain DSL uses stable semantics to describe data objects, transformation operators, filtering, joins, aggregation, UDF/UDTF, and output targets. It does not concern itself with any specific engine or resource configuration.

Below is a simplified core topology of the Chaoxing index Pipeline. This task simultaneously processes real-time product changes, real-time price changes, and offline full product data, forming real-time and offline output branches:

def build_dsl():</code><code>    ctx = SemanticContext(</code><code>        name="chaoxing_hk_pipeline",</code><code>        owner="<owner>"</code><code>    )</code><code>    # Two real-time change streams and one offline full data; specific SQL maintained independently</code><code>    sku_changes = ctx.sql(build_sku_change_stream_sql())</code><code>    price_changes = ctx.sql(build_price_change_stream_sql())</code><code>    full_skus = ctx.sql(build_full_sku_batch_sql())</code><code>    # Real-time branch: merge product and price changes, then merge with wide table state in HBase</code><code>    realtime = (</code><code>        ctx.union(sku_changes, price_changes)</code><code>        .merge("wide_table_chaoxing_hk")</code><code>        .on("rowkey")</code><code>        .alias("realtime_sku")</code><code>    )</code><code>    # Join main site product data to supplement extended attributes needed for index</code><code>    main_skus = ctx.from_table("hbase_main_search_data").alias("main_sku")</code><code>    enriched = (</code><code>        realtime.left_join(main_skus)</code><code>        .on("realtime_sku.sku_id" = main_sku.rowkey")</code><code>        .columns("sku_union_expand_ids")</code><code>    )</code><code>    # Offline branch: full data merged with historical wide table</code><code>    full_snapshot = (</code><code>        full_skus.merge("wide_table_chaoxing_hk")</code><code>        .on("rkey")</code><code>        .alias("full_snapshot")</code><code>    )</code><code>    # Two terminal branches deliver real-time messages and offline index data respectively</code><code>    enriched.to("jdq_search_index_event")</code><code>    full_snapshot.to("search_index_chaoxing_snapshot")</code><code>    return ctx

This example not only connects Source and Sink but also expresses multi-source input, union, state merge, cross-domain left_join, and multi-branch output within a SemanticContext. DSL only describes data dependencies and business topology: which Flink job type the real-time branch falls into, how the offline branch is split, which operators and resources are needed — all are generated by the Sindri backend combining table types and runtime configurations.

4.2 How to Improve DSL Generation Accuracy

The most prominent problem in DSL development today is not "can code be generated" but whether the output is stable and conforms to the current SDK and backend rules.

Incomplete requirement information: When source tables, field mappings, output targets, or run modes are unconfirmed, the model tends to auto-complete business parameters.

Overlapping Skill responsibilities: Requirement extraction, DSL development, fault diagnosis, and submission intervene simultaneously, leading to irrelevant answers or premature submission during development.

Reliance on existing memory for generation: After SDK interfaces, table types, and backend planning rules change, syntax may look correct but topology or execution shape no longer matches the current implementation.

Sindri Plugin does not rely solely on prompt constraints; it breaks the development flow into scenario routing, requirement contract, template selection, and progressive verification. Responsibilities:

Local verification only proves DSL structure is basically sound; final task splitting, engine selection, and resource planning are subject to backend optimization. Submission also adopts a two-phase gate: first generate a preview, only execute real submission after explicit user confirmation. Through the chain of "single-scenario routing → requirement contract → standard examples → SDK facts → local verification → backend planning → human confirmation", DSL accuracy is transformed from a one-shot generation problem into an engineering process that is checkable and rollback-able.

05 ADF: Carrying DAG, Scheduling, State, and Lifecycle

ADF acts as a general execution and control foundation. It receives materialized task requests from the domain backend, organizes DAG nodes and dependencies, manages scheduling, execution, release, and status query, so that upper-layer domain semantics no longer need to repeatedly build these capabilities.

ADF carries task forms such as Spark, Flink, Shell, JDOS, and continuously extends via adaptation mechanisms. The domain backend only describes execution intent; the specific engine on which it lands is decided by the adaptation layer, thereby isolating the diversity of execution engines within ADF.

06 Unified Base Pool: Stable Semantics Connecting Diverse Execution and Delivery

The unified base pool is not about rewriting all tasks into the same code, but establishing a stable middle layer: upstream sources can change, downstream products can change, execution forms can change, but domain semantics, planning interfaces, submission boundaries, and state models remain consistent.

07 Quality, Lineage & Self-Healing: Establish Evidence First, Then Expand Automation

Quality checks, reconciliation, lineage, monitoring, and recovery are indispensable modules for trustworthy data production. The data quality system currently focuses on "making quality assurance solid" and consists of four capabilities:

Data Reconciliation: Full and sampling verification, difference attribution, confirming that multi-source outputs are truly consistent in caliber.

Data Quality: Rule verification, completeness & consistency checks, threshold alerts, turning "whether data meets standards" into measurable constraints.

Lineage Tracking: Field-level lineage, impact analysis, and traceability, letting every artifact answer "where from, who affected".

Monitoring Dashboards: Pipeline health, SLA & timeliness, anomaly observation, presenting runtime trustworthiness in real time.

It should be noted that the data quality system's current positioning is quality assurance — solidifying these four capabilities. The aim is to further evolve into a full-chain trustworthy foundation: opening up the evidence precipitated by quality capabilities for automation to consume gradually.

A judgment on the planning direction is shared here: the self-healing closed loop should not target "full automation" from the start, but should first connect the evidence, then let automation gradually expand within the scope supported by evidence. Reconciliation, quality, lineage, and monitoring produce exactly the evidence foundation for this direction.

Along this direction:

1. Planning Evidence: Record target environment, semantic structure, execution nodes, resource plans, and confirmation results.

2. Submission Evidence: Record preview, permission judgment, submission results, and Sindri & ADF object identifiers.

3. Runtime Evidence: Return DAG, tasks, instances, logs, and structured errors, allowing the Agent to continue handling based on facts.

On these foundations, local verification, diagnostic suggestions, and limited retries can be gradually integrated. True self-healing still requires stable state machines, idempotency mechanisms, risk grading, retry limits, result re-verification, and human takeover — this is why "evidence first, automation later" is safer: the more complete the evidence chain, the more actions can be safely handed to automation.

08 Partial Results Showcase

Task production only needs a description document or a prompt to generate a base pool task; the corresponding execution chain is automatically generated, and key data is strictly gated during execution to avoid user errors.

After task generation, corresponding execution files are produced; upon user submission, resource archiving is synchronized for subsequent iterative development and overall system backtesting.

In the data quality system, base pool task lineage is automatically drawn (currently synchronously drawing other tasks associated with each stage), and clicking a stage synchronously displays detailed information under that stage, facilitating full-chain tracing and troubleshooting.

09 Conclusion

Data Nirvana is not adding a chat entry to the old chain, but reshaping the production method from business intent to data value.

Forge Agent serves as the unified AI facade, organizing requirements, knowledge, processes, and feedback; Sindri Plugin/MCP turns deterministic capabilities into discoverable, composable tools; Sindri Backend guards the planning and submission boundaries; ADF handles DAG, scheduling, state, and lifecycle. Together they form a collaboration chain of "intelligent organization, deterministic planning, controlled submission, stable execution".

When responsibility boundaries are clearly drawn, when every planning has basis, every submission has confirmation, every run has state, AI becomes not just a content generator but a trustworthy data productive force.