How JD Retail Automates AB Experiment Data Pipelines with Data Weaving

1. Challenges in AB Experiment Scenarios

AB testing relies on controlled variable methods to compare metric differences between groups, but faces three core data challenges: consistency of metric definitions across consumption contexts, scientific validation of metric distributions (including p‑values and confidence intervals), and timeliness when handling thousands of daily experiments that demand efficient scheduling and delivery.

Consistency Requirement: Metric definitions (aggregation, data type, source tables) must remain identical whether used in experiment analysis or BI dashboards to avoid manual alignment.

Scientific Challenge: Experiments need statistically comparable samples; noise from business logic or algorithmic changes must be mitigated by controlling Type I/II error rates and adding denoising logic.

Timeliness Issue: Large‑scale experiments generate massive task volumes; manual development and custom orchestration cannot sustain the required delivery speed.

To address these, JD Retail seeks an automated solution that aligns metric definitions across contexts and builds the entire data chain automatically.

2. Data Weaving Management Concept

Data weaving introduces a virtualized, logical data integration layer that abstracts physical assets into reusable semantic models. The architecture consists of four layers:

Data Asset Layer – traditional data warehouse tables (fact and dimension assets).

Data Virtual Layer – semantic model entry point where users define logical entities (metric market, dimension center, source management).

Data Materialization Layer – on‑demand physical integration of assets, applying degradation strategies as needed.

Unified Data Service Layer – provides consistent data outputs for multiple consumption scenarios.

3. Technical Details of AB Automation

3.1 Overall Architecture

Traditional AB experiment delivery requires manual mapping of business requirements to data models, joining fact tables with experiment split tables, and configuring platform settings, leading to high manual effort and error risk.

With data‑weaving automation, the system automatically generates the necessary joins and logical tables based on metric definitions, expands them for daily and cumulative calculations, and injects statistical calculations (sample sum, sum of squares, covariance) required for p‑values and confidence intervals.

3.2 Metric Language

Metrics are categorized into derived metrics (primary and sub‑metrics) and composite metrics (arithmetic combinations). For example, average contribution amount = total transaction amount ÷ number of transacting users, with optional modifiers (e.g., new‑product filter) that are treated as virtual decorations rather than permanent schema changes.

3.3 Metric Decomposition

Business language is translated into data‑weaving elements: who, what, when, where, and how. This enables automatic generation of logical tables, split by experiment ID, and supports both daily and cumulative aggregation strategies.

3.4 Logical Modeling & Data Acceleration

Logical tables are first widened (adding necessary dimensions) and then heightened (adding experiment‑specific fields). The system performs on‑demand materialization, creating only the required schema fragments for each experiment, reducing storage and compute overhead.

3.5 Smart Orchestration & SQL Engine

Experiments generate multiple DAG nodes for different time windows (daily, cumulative) and data granularity (raw, trimmed, light‑aggregate, pre‑computed). The orchestration layer dynamically merges tasks across experiments that share the same logical table, dramatically reducing the number of executed jobs.

3.6 Composite Metric DAG

Composite metrics are split into numerator and denominator, routed to appropriate logical tables (order, exposure, click), and then processed with time‑window strategies. This results in a scalable pipeline that can handle thousands of concurrent experiments.

4. Current Progress and Future Outlook

At present, 60% of metrics can be auto‑computed via experiment subscriptions, delivering results in seconds and reducing delivery cycles from weeks to days. Future work focuses on expanding metric coverage, improving latency and performance for large‑scale usage, enhancing experiment flexibility, and simplifying troubleshooting.

5. Q&A Highlights

When multiple experiments share metrics, the system merges their logical tables by adding an experiment‑ID list filter, turning many separate tasks into a single unified job, which is crucial for handling the massive task explosion in large enterprises.