Skip to content
Batch Processing Scheduling Architecture: Reliability Beyond Scheduled Triggers

Batch Processing Scheduling Architecture: Reliability Beyond Scheduled Triggers

This chapter distinguishes scheduling from batch execution, covering single-machine, cluster, distributed, and pipeline batch patterns, emphasizing reliability through state tracking, idempotency, sharding, checkpoints, and explicit dependencies over fixed timing. It also outlines practical platform capabilities like versioned task definitions, unique run instances, state machines, leases, business idempotency keys, resource quotas, and end-to-end reconciliation.

Cilantro Loves Linux
Cilantro Loves Linux
Cilantro Loves Linux

Chapter Overview

Chapter 11 of Architecture Fundamentals: From Requirements to Architecture by Yin Hongliang focuses on batch processing scheduling architecture. The core insight is that a task starting on time and exiting normally does not guarantee all data was processed correctly. The chapter separates scheduling (when, with what parameters, and under what dependencies to run) from execution (the actual data processing). In batch contexts, single-machine suits low-complexity tasks; primary-backup or cluster deployments mainly handle executor node failures; sharding parallelism shortens processing windows; and pipelines express complex dependencies.

Batch reliability depends not only on task startup but also on avoiding duplication, preventing omissions, knowing where to resume after failure, and ensuring output completeness. State recording, idempotency, sharding, checkpoints, and dependency management matter more than cron expressions.

Key Concepts

11.1 Batch Processing vs. Scheduling

Tasks can be triggered by time or events. Batch processing fits large data volumes, low real-time requirements, and clear execution boundaries. A simple loop to estimate duration often reveals whether a single-threaded approach is acceptable.

A task model must describe at least trigger conditions, parameters, executor, timeout, retry policy, concurrency strategy, and dependencies. A run instance records start, end, status, processed volume, errors, and log location.

11.2 Single-Machine Batch Processing

Single-machine solutions suit limited-scale tasks. Scheduling definitions should be separated from execution code so that timing, parameters, and start/stop can be managed dynamically. Serial implementation is simple and order-explicit; parallelism improves throughput but introduces thread safety, result aggregation, and downstream capacity concerns.

Parallelism should be determined by per-item latency, data volume, and resource bottlenecks. Internal thread pools must have bounded queues and overload policies to prevent batch jobs from contending with online services for all resources.

11.3 Cluster Batch Processing

Cluster mode improves availability via registration, heartbeats, and leader election. Typically only one leader executes a given task, so compute capacity does not increase. When the leader fails, the new node must resume from the business state of unfinished data, not re-execute from the beginning.

Election only decides who is eligible to run; leases, fencing tokens, or database constraints are still needed to prevent two nodes from simultaneously believing they are the leader during network partitions.

11.4 Distributed Batch Processing

Distributed mode partitions data across multiple worker nodes for parallel processing. Sharding can be done by primary key range, time, hash, or pre-generated task items. The scheduling center assigns shards; executors claim, renew leases, process, and report results. Failed shards are taken over by other nodes.

Sharding must avoid duplication and omission. Hot shards need support for re-splitting or dynamic load balancing. Final results may require multi-level merging, which must handle partial success, retries, and idempotent aggregation.

11.5 Pipeline Batch Processing

Complex tasks should not rely on fixed time offsets. Instead, explicitly model upstream/downstream dependencies, parallel branches, conditional branches, and aggregation nodes so downstream tasks trigger upon upstream success. This prevents overlaps and data gaps when runtimes vary.

Pipelines also need failure propagation rules: whether a node failure stops the entire chain, is skipped, retried, or triggers compensation — all driven by business semantics.

Architecture Diagrams

The first diagram combines scheduling, sharding, and recovery. The second illustrates how pipelines are driven by dependency relationships rather than human-estimated fixed times.

Mermaid diagram: scheduling, sharding, and recovery
Mermaid diagram: scheduling, sharding, and recovery
Mermaid diagram: pipeline dependency-driven execution
Mermaid diagram: pipeline dependency-driven execution

Personal Insights

"Scheduling success does not equal business completion" is the key takeaway. An artifact cleanup job may start on time but fail to delete all intended artifacts or may accidentally delete versions still referenced by releases. A compliance scan may exit normally yet not cover all projects. A green process status and a correct business result are separated by a missing completion standard.

Therefore, the author now first defines how many items to process, how many succeed, what is skipped, what fails, and whether re-runs cause side effects — before discussing cron expressions and execution nodes. The book's distinction between cluster and distributed batch processing fits specific scheduling scenarios better than as universal definitions: failure takeover, shortening processing windows, and expressing task dependencies are three different problems.

Practical Implementation Considerations

Task Platform Capabilities

Separate task definition from execution code; support versioning, parameters, enable/disable, and permission management.

Generate a unique run instance per execution, recording trigger source, input range, and code version.

Use explicit state machines, leases, checkpoints, and retry limits for both tasks and shards.

Equip write operations with business idempotency keys to prevent duplicate processing during failover.

Explicitly define task dependencies, conditional branches, parallelism, and aggregation relationships.

Set resource quotas to prevent offline jobs from impacting online services.

Perform end-to-end reconciliation on total processed count, success count, failure count, and business aggregates.

Capacity Estimation

Calculate minimum throughput from data volume, average per-item latency, and target completion window. Then determine reasonable node count based on database and external channel capacity. Load testing must include failure retries and downstream rate limiting, not just the happy path.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

distributed systemsShardingbatch processingSchedulingreliabilityIdempotencyPipelinecheckpointing
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.