Data Warehouse Development Standards: Phase Planning, Roles, and End-to-End Process

Overview

The data warehouse development standard provides a structured workflow to improve efficiency, ensure orderly execution, and reduce cost and risk. It defines the phases, roles, and overall process for data product managers, designers, developers, testers, and operations personnel.

Phase Planning

Requirement Phase: How data product managers handle evolving business needs.

Design Phase: How designers balance performance, cost, efficiency, and quality to organize and store data.

Development Phase: How developers code efficiently and consistently.

Testing Phase: How testers expose code issues and project risks to improve output quality.

Release Phase: How qualified programs are smoothly released to production.

Operations Phase: How ops staff ensure timeliness and stability of data output.

Role Responsibilities

Data Product Manager : Capture and evaluate business data needs, lead requirement reviews, produce requirement documents, and oversee technical reviews.

Designer : Perform data profiling, understand data quality and distribution, and create table, mapping, and scheduling designs.

Developer : Implement code based on design artifacts, conduct unit testing, and participate in code reviews.

Tester : Verify consistency between requirements and results, identify code defects and project risks.

Operations Engineer : Handle release tasks, address data, program, scheduling, and monitoring alerts, ensuring timely and stable data production.

Information Security & Compliance : Assess security and compliance before requirement review.

Overall Process Diagram

Requirement Phase

Initial Requirement Process

When a business unit first raises a need, the data product manager evaluates technical, data, and compliance feasibility, drafts a product requirement document, and organizes a multi‑party review meeting to finalize the implementation plan.

Steps:

Submit Requirement

External communication led by the product manager to understand business scenario, goals, and value.

Note: No discussion of implementation details at this stage.

Draft the initial product requirement document using the standard template.

Analyze Requirement

Reasonableness assessment.

Data feasibility: can existing data support the need? If not, plan data extraction.

Technical feasibility: can current data models support the need? If not, plan model changes and test in a sandbox.

Security & compliance: control data flow, evaluate external data export against policies.

Feasibility Review

Product manager leads a review with designers and security staff.

Requirement Review Meeting

All parties raise questions on the document; reach consensus on solutions.

Note: No critical issues may remain, otherwise the review fails.

Confirm Requirement

If no objections within N working days, the document is finalized and the project moves to design.

Iterative Requirement Process

After the initial requirement is approved, any subsequent changes follow a similar feasibility analysis and detail design. The product manager records changes in the iteration request form and may use a platform‑plus‑database approach to version requirements.

Steps:

Request Change: Record new needs in the iteration request form.

Review Change: Usually a meeting led by the product manager; for minor changes, email or on‑site review may suffice. The review still covers technical, data, security, and compliance feasibility.

Confirm & Merge: Merge the new details into the existing requirement document; if no objections within two days, the change is confirmed.

Design Phase

After requirements are finalized, designers conduct data profiling and system design.

Data Exploration : Assess data quality, distribution, null/abnormal values, relationships, formats, and incremental rules. Produce a data exploration report; if data cannot support the requirement, trigger an iterative requirement.

System Design (Table, Mapping, Scheduling)

Table Design: Define table names, field names, types, comments, and security levels; ensure consistent naming across tables.

Mapping Design: Document field generation logic, table relationships, and transformation algorithms.

Scheduling Design: Define node dependencies (one node produces one table), upstream/downstream data flow, parallelism, execution windows (hourly, daily, weekly, etc.), baseline alerts, priority settings, and data flow constraints (ODS→DWD→DWS→ADS only).

Development Phase

Developers translate design artifacts into code, ensuring coding standards, accuracy, and unit testing.

Code Development

Unit Testing

Code Review

Testing Phase

After development, testing validates code, uncovers defects, and assesses project risk.

Test Analysis

Prepare Test Cases

Execute Tests

UAT: Product manager conducts acceptance testing from a business perspective and issues an acceptance report.

Release Phase

Qualified programs are deployed to production following normal or emergency release procedures.

Normal Release : Predictable, scheduled releases with prior planning.

Emergency Release : Rapid response to critical bugs or urgent business needs; may require immediate approval if a normal window is unavailable.

Release steps include release request, approval, and execution.

Operations Phase

Post‑release, operations staff monitor and resolve exceptions in data, programs, scheduling, and alerts to ensure timely and stable data output.

Handle program exceptions and performance tuning.

Address scheduling anomalies.

Analyze and optimize data quality monitoring rules.

Investigate data anomalies.

Operations workflow: analyze impact → devise and implement solution → verify implementation.