BestHub
Sign in
Tagged articles
11 articles
Page 1 of 1
JavaEdge
JavaEdge
Feb 20, 2024 · Big Data

Designing a Scalable Data Quality Center for Offline Big‑Data Pipelines

This article describes the design and implementation of a platform‑wide Data Quality Center for offline big‑data pipelines, covering research of existing solutions, design goals, system architecture based on DolphinScheduler, rule definition language, binding and execution mechanisms, and future enhancements such as lineage monitoring and real‑time checks.

Apache GriffinBig DataData Quality
0 likes · 18 min read
Designing a Scalable Data Quality Center for Offline Big‑Data Pipelines
DataFunTalk
DataFunTalk
Apr 4, 2023 · Big Data

Compass: An Open‑Source Big Data Task Diagnosis Platform for DolphinScheduler, Airflow and Spark

Compass is an open‑source big‑data diagnostic platform developed by OPPO that provides non‑intrusive, real‑time monitoring and root‑cause analysis for offline and streaming tasks on schedulers such as DolphinScheduler and Airflow, covering workflow‑level failures, Spark engine anomalies, resource usage, and offering one‑click reports and extensible rule‑based diagnostics.

Big DataDolphinSchedulerSpark
0 likes · 13 min read
Compass: An Open‑Source Big Data Task Diagnosis Platform for DolphinScheduler, Airflow and Spark
DataFunSummit
DataFunSummit
Mar 6, 2023 · Big Data

Building a Unified Scheduling Center with Apache DolphinScheduler: Lenovo’s Practice

This article details Lenovo’s implementation of a unified scheduling center using Apache DolphinScheduler, covering background requirements, reasons for choosing the platform, architectural evolution, feature enhancements, and practical deployments such as HTTP task parameter passing, Java task plugins, global parameters, and future roadmap.

DolphinSchedulerLenovobigdata
0 likes · 19 min read
Building a Unified Scheduling Center with Apache DolphinScheduler: Lenovo’s Practice
Programmer DD
Programmer DD
Mar 2, 2023 · Backend Development

Why DolphinScheduler Is the Next Powerhouse for Distributed Task Management

DolphinScheduler is an open‑source distributed task scheduling system that supports multiple task types, offers visual workflow orchestration and monitoring, and scales to thousands of servers, making it a robust solution for backend and big‑data processing scenarios.

Big DataDistributed SchedulingDolphinScheduler
0 likes · 4 min read
Why DolphinScheduler Is the Next Powerhouse for Distributed Task Management
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 26, 2022 · Backend Development

DolphinScheduler Design, Architecture, and Source Code Analysis

This article provides a comprehensive overview of DolphinScheduler’s design strategies, distributed architecture, fault‑tolerance mechanisms, configuration files, core APIs, Quartz integration, master‑worker execution flow, RPC communication, load‑balancing algorithms, and logging services, accompanied by detailed code excerpts and diagrams.

Backend DevelopmentDistributed SystemsDolphinScheduler
0 likes · 46 min read
DolphinScheduler Design, Architecture, and Source Code Analysis
Big Data Technology Architecture
Big Data Technology Architecture
Sep 18, 2022 · Backend Development

Design and Source Code Analysis of Apache DolphinScheduler

This article provides an in‑depth technical overview of Apache DolphinScheduler, covering its distributed design strategies, fault‑tolerance mechanisms, remote log access, source‑code module breakdown, API interfaces, Quartz integration, master‑worker execution flows, RPC communication, load‑balancing algorithms, logging services, and community contribution guidelines.

Distributed SchedulingDolphinSchedulerLog Service
0 likes · 47 min read
Design and Source Code Analysis of Apache DolphinScheduler
DataFunTalk
DataFunTalk
May 31, 2022 · Artificial Intelligence

Using DolphinScheduler OpenMLDB Task for End‑to‑End MLOps Workflow

This article introduces the DolphinScheduler OpenMLDB Task, explains how it integrates OpenMLDB's feature platform into DolphinScheduler workflows to create a complete MLOps pipeline, and provides a step‑by‑step demonstration using the TalkingData ad‑fraud detection dataset from Kaggle.

DolphinSchedulerMLOpsOpenMLDB
0 likes · 7 min read
Using DolphinScheduler OpenMLDB Task for End‑to‑End MLOps Workflow
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 14, 2022 · Big Data

Comparison of Common Big Data Scheduling Systems: Oozie, Azkaban, Airflow, XXL‑Job, and DolphinScheduler

This article provides a comparative overview of several popular big‑data workflow schedulers—including Oozie, Azkaban, Airflow, XXL‑Job, and DolphinScheduler—detailing their supported task types, visual workflow definition, monitoring capabilities, pause/resume features, high‑availability options, and other notable characteristics.

AirflowDolphinSchedulerOozie
0 likes · 9 min read
Comparison of Common Big Data Scheduling Systems: Oozie, Azkaban, Airflow, XXL‑Job, and DolphinScheduler
DataFunTalk
DataFunTalk
Mar 3, 2022 · Big Data

Youzan Data Platform and DP Data Development Platform: Architecture, Core Modules, and Scheduling System Upgrade

This article presents an in‑depth overview of Youzan's data platform, introduces the DP data development platform with its key features and workflow, details the core module architecture—including service, scheduling, and component layers—and explains the migration from Airflow to DolphinScheduler to improve performance, stability, and scalability.

Big DataData DevelopmentData Platform
0 likes · 14 min read
Youzan Data Platform and DP Data Development Platform: Architecture, Core Modules, and Scheduling System Upgrade
政采云技术
政采云技术
Feb 24, 2022 · Backend Development

Implementation of Workflow Versioning in DolphinScheduler

This article explains how DolphinScheduler (DS) implements workflow versioning by introducing integer‑based versions, describes the core principles of workflow and task relationships, outlines the key database tables and architecture, and provides detailed Java code for creating, updating, and managing workflow definitions, tasks, and their relations.

BackendDolphinSchedulerVersioning
0 likes · 13 min read
Implementation of Workflow Versioning in DolphinScheduler