DataOps Best Practices on Alibaba Cloud: Architecture, CI/CD, Quality & Self-Healing

Core Architecture Design: Toolchain Collaboration

The article presents a four-layer technical stack for DataOps on Alibaba Cloud:

Bottom layer – Big Data Lakehouse Foundation: Storage and compute via MaxCompute (closed-source), OSS, and open-source big data components.

Second layer – Full Lifecycle Development & Operations: DataWorks and Yunxiao pipeline (or Jenkins) integrate to control big data task development, testing, and release; DataWorks Operations Center provides scheduling task monitoring.

Third layer – Development Governance: DataWorks capabilities for metadata management, data quality rules, and lineage.

Top layer – Data Applications: BI analysis, data products, and services.

A mapping aligns each module with specific tools and core functions:

Data Integration: DataWorks Data Integration – supports 80+ sources (MySQL, OSS, Kafka), config-driven full/incremental sync including Flink CDC.

Stream/Batch Processing: Flink + MaxCompute – Flink handles real-time streams (millisecond latency), MaxCompute handles batch jobs (TB-scale offline compute).

Real-time Analytics: StarRocks – sub-second interactive queries, reads MaxCompute/Paimon directly, avoiding data movement.

Development & Operations: DataWorks Data Development – visual SQL/Flink job development with integrated scheduling, monitoring, version management.

CI/CD: Yunxiao Pipeline – code hosting, automated build, test, deploy, GitOps full-process management.

Metadata & Quality: DataWorks Metadata Center – auto-collects lineage, configures quality rules (null checks, uniqueness constraints).

Data Development Best Practices (30%+ Efficiency Gain)

1. Standardized Development Process (DataWorks)

Data Modeling & Asset Accumulation

Unified planning of data models enables shared common models, eliminates data silos, improves standardization, reusability, and extensibility. Dimensional Modeling: Design star/snowflake schemas in DataWorks Data Model module, auto-generate MaxCompute DDL, avoiding duplicate development. Common Layer Construction: Universal metrics (e.g., total users, order count) sunk into Common Dimension Model (CDM), exposed via Data Service API calling StarRocks data.

Stream/Batch Task Development Paradigm

Flink (real-time) complements MaxCompute (batch) under stream-batch unification. A unified paradigm ensures efficiency and consistency.

Flink Real-time Tasks: Develop with Flink SQL , configure sources (Kafka), processing logic (window aggregation), sinks (Paimon/StarRocks) via DataWorks visual UI. Example: compute user visit frequency within 30 minutes, write results to StarRocks for dashboard display.

MaxCompute Batch Tasks: Use SQL templates to rapidly generate warehouse layered tasks (ODS→DWD→DWS), auto-inherit upstream table dependencies. Configure dynamic partitioning and bucketing to improve scan efficiency (e.g., partition by dt, bucket by user_id).

Code Management Standards

Version Control: All SQL/Flink code hosted in Yunxiao Codeup; branch strategy Develop → Release → Prod isolates environment changes.

Code Review: Merge requests trigger mandatory data quality scans and lineage impact analysis to prevent dirty data entering production.

Automated CI/CD Pipeline Design (Yunxiao Pipeline)

Pipeline Stages

Regular Development Flow: Developers commit to develop branch → testers run unit tests and functional data tests → failures feed back for fixes → code scan with rule warnings → approver confirms → deploy to UAT for pre-production validation (response speed, compute correctness, concurrency) → UAT pass promotes to production; failures return to developers.

Emergency Hotfix Flow: Post-release bugs trigger hotfix branch from main → emergency code review → direct production deploy.

Branch Definitions: develop is main integration branch for daily work; hotfix is temporary for critical production fixes, merged back to main and develop after fix.

Key Implementation Details

Automated Testing:

Unit Tests: Use MaxCompute Local Mode to simulate offline tasks, assert SQL output correctness.

Integration Tests: Spin up Flink mini-cluster, verify end-to-end latency (≤500ms) and Exactly-Once semantics.

Gray (UAT) Release:

New tasks deploy to shadow tables (e.g., dws_order_analysis_gray), dual-write to StarRocks for data consistency comparison.

Traffic gradually shifted (1% → 10% → 100%), monitor query performance; auto-rollback if 95th-percentile latency exceeds 1s (e.g., QPS spike).

Data Quality & Monitoring System (80%+ Self-Healing Rate)

Full-Chain Quality Control

Ingestion Layer: DataWorks Data Integration tasks configure field-level validation (e.g., order amount ≥0); dirty data auto-written to ods_error_log and triggers DingTalk alert.

Processing Layer:

Flink jobs set Checkpoint timeout threshold (e.g., 30s); 3 consecutive failures auto-trigger task restart.

MaxCompute tasks enforce strict dependency checks – block current task if upstream table not produced on time.

Output Layer: StarRocks analytical results compared against offline T+1 reports; discrepancy >5% triggers manual investigation.

Intelligent Monitoring & Alerting

Metrics Monitoring: Prometheus + Grafana monitor StarRocks QPS, Flink Checkpoint size, MaxCompute queue length. Example: StarRocks query latency 95th percentile >2s for 10 consecutive minutes triggers automatic BE node scaling (combined with K8s elastic scaling).

Log Tracing: Full-chain logs ingested into SLS Log Service ; TraceID correlates Flink task ID, MaxCompute job ID, StarRocks query ID. Enables rapid fault location – e.g., report data anomaly traced to Flink deserialization error within 3 minutes.

Collaboration & Efficiency (Cross-Team Synergy)

Role Division & Permission Management

Data Developer: Core Tools: DataWorks + Yunxiao. Responsibilities: Build data pipelines, commit to Develop branch, write unit test cases.

Business Analyst: Core Tools: StarRocks + Data Visualization. Responsibilities: Fetch metrics via DataWorks Data Service, ad-hoc query & multi-dimensional analysis in StarRocks.

Data Operations: Core Tools: Yunxiao + K8s + CloudMonitor + SLS. Responsibilities: Maintain pipeline stability, optimize component performance (Flink parallelism tuning, StarRocks index design); configure alerts, handle production incidents, manage resource quotas (MaxCompute CU allocation).

Self-Service Capabilities

Data Catalog: DataWorks auto-generates data map; search table names, field descriptions, business definitions – reduces communication overhead.

Task Template Library: 20+ predefined templates (e.g., real-time UV calculation, offline slowly changing dimension tables); business users generate tasks via visual UI with one click.

Performance Optimization & Cost Control

Compute Resource Optimization

Flink: Use incremental processing instead of full recomputation (e.g., RocksDB state backend reduces memory footprint); schedule non-real-time tasks in off-peak windows (e.g., 02:00–06:00) to avoid contention with real-time jobs.

MaxCompute: Adjust storage class by access frequency – data untouched >6 months moved to cold storage , cutting cost >60% vs standard storage; enable elastic compute – scale out for nightly batch peaks, scale in for daytime.

StarRocks Storage-Compute Separation: Tier by query heat – high-frequency tables enable cache acceleration ( "datacache.enable" = "true"), low-frequency disable ( "datacache.enable" = "false"); dimension tables use local dictionary encoding , high-cardinality fact tables use columnar storage + aggregate indexes .

Security & Compliance Practices

Access Control: RAM policies restrict DataWorks developer roles to read-only ods_* tables, prohibit direct dws_* core table access. Production data access requires approval flow with time-limited (days/hours) table-level grants.

Dynamic Masking: StarRocks queries on sensitive fields (e.g., phone) auto-apply masking functions (e.g., mask_phone(phone)).

Operation Audit: Log DataWorks task releases, MaxCompute resource consumption, StarRocks query logs – meets Grade 3 Cybersecurity Protection (等保三级) audit requirements.

Implementation Roadmap (Phased Rollout)

Phase 1: Foundation (1–3 months)

Integrate DataWorks with Yunxiao pipeline; achieve CI/CD for MaxCompute and Data Integration tasks.

Deploy Flink real-time tasks to production, covering 20% core scenarios (e.g., real-time order monitoring).

Phase 2: Efficiency Boost (3–6 months)

Build common dimension models, sink 50+ universal metrics, reduce duplicate development .

Launch data quality monitoring system, auto-intercept >90% dirty data ingress.

Phase 3: Intelligent Operations (6–12 months)

Introduce AIGC-assisted data development (auto-generate SQL templates, smart index recommendation).

Achieve self-healing (Flink task failure auto-retry + resource scaling), reduce MTTR (Mean Time To Repair) to <15 minutes.

DataOps Value Realization

Development Efficiency: Task delivery cycle shortened from weeks to hours; code reuse rate up 40%.

Data Quality: Error rate dropped from 0.5% to 0.05%; report verification time cut 80%.

Cost Optimization: Compute cost down 30%, storage cost down 50% (cold-hot tiering + smart compression).

Business Responsiveness: Real-time analytics latency ≤1s, enabling second-level decisions (real-time risk control, promo campaign real-time effect analysis).

Ultimately forms a "standardized development, automated operations, controllable quality, efficient collaboration" DataOps system, providing a solid foundation for enterprise data-driven business innovation.