Zero‑Code Real‑Time Data Pipelines Powered by an AI‑Driven Metric Engine

Background

Live‑streaming scenarios require massive real‑time data processing such as 360 promotion analysis, real‑time leaderboards, and streaming feature pipelines for ads and policy control. Compared with batch jobs, streaming development must handle windows, state, time semantics and out‑of‑order data, making the development cost and review effort much higher.

Metric‑Driven Paradigm

The team abstracts the core business logic of a data link as "dimension + metric". Users write a natural‑language description (≤200 characters). An AI model extracts the target metric, its dimensions, and window semantics, then generates a structured DSL which is translated into Flink SQL. This reduces the development cycle from days to minutes.

1. Requirement Description

Requirement clarification : the system automatically retrieves the metric, dimension and window information from the user’s natural‑language input.

Requirement confirmation : after the user verifies the extracted information, the system produces the DSL.

2. DSL Confirmation

The user reviews the DSL (a shared contract among front‑end, AI and back‑end) and confirms it. The system then generates the corresponding Flink SQL. The DSL acts as a consensus contract that bridges the three components.

3. SQL Generation

Through a DSL‑to‑SQL translator, the engine produces a Flink SQL script with 100 % accuracy, including source subscriptions, temporary view definitions, aggregations and target sinks.

4. Incremental Adjustment (Optional)

If the initial SQL does not fully satisfy the requirement, the user can describe additional adjustments (e.g., add a filter in a temporary view). The system creates a Hook protocol – an incremental DSL patch – and re‑generates the SQL without altering the overall topology.

{
  "hooks":[
    {
      "hookType":"FILTER | DIMENSION_TRANSFORM",
      "layerType":"SOURCE | VIEW | TARGET",
      "nodeId":"source_001 | tmp_view_001 | target_001",
      "mergeStrategy":"REPLACE | APPEND_AND | APPEND_OR",
      "filterExpression":"...",
      "dimensionName":"...",
      "expression":"...",
      "reason":"..."
    }
  ]
}

Engine Core Actions

Dependency back‑trace : starting from the target metric, the engine walks back through all dependent metrics to automatically construct the full DAG topology.

Dual‑path dimension filling : process dimensions referenced by metrics (process dimensions) and output dimensions (aggregation granularity) separately, ensuring they do not interfere.

Design Principles

Metric‑driven as the core development paradigm.

Separate "understanding" (LLM) from "correctness" (deterministic engine) to limit AI hallucination.

DSL as a universal contract for storage, transmission and validation across front‑end, AI and back‑end.

Hook protocol as a pluggable incremental mechanism, with pre‑ and post‑validation fences to guarantee topology consistency.

Benefits

Users type a concise natural‑language description, the AI handles parsing, the engine builds a topology, safety fences ensure no broken links, and the final Flink job can be published to the Dataphin platform with a single click. Development time shrinks to minutes, and metric assets become reusable across tasks, forming a virtuous cycle of efficiency.