Turn Daily Batches into Re‑runnable Workflows with DolphinScheduler (Big Data Series 8)

Background and Positioning

When moving a data lakehouse to production, the bottleneck is not writing a single SQL but orchestrating daily batch jobs: scheduling times, automatic retries, handling incomplete ODS partitions, and backfilling specific dates while preserving dependency chains.

Why manual scripts are insufficient

Uncoordinated cron jobs lead to missed dependencies.

Failure handling requires manual intervention.

Backfilling often involves ad‑hoc script changes that can omit steps.

Apache DolphinScheduler provides a DAG‑based workflow engine that triggers jobs on a calendar, enforces dependencies, supports retries, alerts, and backfill capabilities.

Core Concepts

Project, Workflow, Task, Instance

Project : logical boundary for resources and permissions, e.g., an offline data‑warehouse project.

Workflow definition : a DAG blueprint describing tasks and their connections.

Task : a node that can be a Shell script, SQL, Spark job, HTTP call, etc.

Instance : a concrete execution of a workflow for a specific business date.

Think of a workflow definition as a recipe and an instance as the actual meal prepared on a given day.

Component Collaboration (Simplified)

API : login, create projects, store workflows, accept start requests.

Master : parses the DAG, schedules tasks, checks dependency satisfaction.

Worker : executes the actual Shell/Spark/HTTP commands.

Alert : sends failure or timeout notifications (configuration omitted for brevity).

Registry/Meta‑DB : service discovery and persistence (H2 embedded in standalone mode).

The standalone mode packs all roles into a single process with an embedded H2 database, suitable for learning.

Typical Offline Batch DAG

The demo builds three Shell tasks that simulate real Hive/Spark/ADS jobs: ods_check: writes a marker file to indicate ODS partition readiness. spark_like_etl: reads the marker and writes a DWS file, mimicking Spark/Hive processing. ads_touch: writes a ready flag for the ADS layer, optionally performing a ClickHouse health check.

Dependencies are expressed as ods_check → spark_like_etl → ads_touch with a global bizdate parameter representing the business date for backfill scenarios.

Repository Layout

Docker image: apache/dolphinscheduler-standalone-server:3.2.2 UI URL: http://127.0.0.1:12345/dolphinscheduler/ui Credentials: admin / dolphinscheduler123 Demo scripts mounted from host docker/09-dolphin/demo/scripts to container

/opt/soft/demo

Local Environment and Smoke Test

./scripts/up.sh 09-dolphin
./docker/09-dolphin/scripts/wait-ready.sh

Open the UI, log in, and verify health with:

curl -sS 'http://127.0.0.1:12345/dolphinscheduler/actuator/health'
./docker/09-dolphin/scripts/api-login.sh

Hands‑on Experiment: First Offline Workflow

Demo chain description

ods_check

( 01_ods_check.sh): simulate ODS partition readiness by writing a marker. spark_like_etl ( 02_spark_like_etl.sh): simulate Spark/Hive daily batch by reading the marker and writing DWS. ads_touch ( 03_ads_touch.sh): simulate ADS touch by writing a ready flag; optional ClickHouse probe.

One‑click creation and launch

chmod +x docker/09-dolphin/scripts/*.sh docker/09-dolphin/demo/scripts/*.sh
./docker/09-dolphin/scripts/create-offline-workflow.sh

The script logs in, creates a project bigdata-offline if absent, generates task code, creates a workflow daily_offline_etl, and marks it ONLINE.

Run a single instance

./docker/09-dolphin/scripts/run-workflow.sh

To backfill a specific date, run:

./docker/09-dolphin/scripts/run-workflow.sh '' '' 2026-07-30

Successful UI instances show three tasks completing in order, with output files under /tmp/dolphin-demo/.

Mapping to real Hive/Spark jobs

Shell + marker → Hive partition check (MSCK) / SHOW.

Shell writing DWS → Spark spark-submit of a layering JAR or SQL task.

Shell triggering ADS → Doris Stream Load or ClickHouse HTTP INSERT.

Key takeaway: DAG edges express dependencies, parameters convey business dates, and instances represent concrete runs. Workers must have appropriate clients and network access.

Intuition for Dependency, Retry, Backfill, Alert

Dependency : downstream tasks do not run if upstream fails, preventing dirty ADS data.

Retry : configure retry count and interval to survive transient glitches.

Backfill : re‑run historic dates by passing bizdate; ensure idempotent ETL logic.

Alert : failures or timeouts push notifications to email/DingTalk (requires extra config).

When backfilling, the same bizdate must produce identical results; otherwise, stronger scheduling leads to more dirty data.

Common Pitfalls and Troubleshooting

UI not reachable or health check fails – check container logs, memory limits, and wait a couple of minutes.

"parameter not valid" on workflow creation – ensure tasks include isCache and version fields; use the provided script.

Shell script not found – verify the demo scripts are mounted and list /opt/soft/demo inside the container.

Missing marker in downstream – upstream may have failed or bizdate mismatch; inspect task logs.

Definitions disappear after down – standalone H2 is non‑persistent; rerun the create script.

Master overload – memory contention; stop other large modules first.

Real Spark jobs fail – the standalone image lacks cluster clients; customize the worker environment.

Position in the Big Data Pipeline

Kafka / source DB → Ingestion → (DolphinScheduler) Spark/Hive daily batch → Doris/ClickHouse ADS
                              ↘ retry / daily backfill

Scheduling turns already‑working jobs into a stable daily production output.

Summary

DolphinScheduler governs when a workflow runs, whether dependencies are satisfied, and how failures are handled; it does not replace Spark/Hive.

Key objects: Project → Workflow definition → Task DAG → Execution instance.

Use a Docker standalone for rapid hands‑on; create an ODS → ETL → ADS Shell chain via Open API, and explore dependency, retry, and backfill concepts.

Code and environment are available at https://gitcode.com/qq_37953312/big-data.