Designing a Robust Batch Processing Module: Key Architecture Insights

When a system module experiences performance bottlenecks due to massive batch jobs, isolating those jobs with a batch‑processing framework becomes necessary. While Spring Batch provides a well‑packaged solution, many fundamental design issues still need attention to ensure stable online transaction performance.

1 Application Layer

The classic Java batch framework is Spring Batch. Instead of diving into low‑level details, this section highlights the key concerns and possible mitigation strategies.

Batch Framework Considerations

Task scheduling – typically integrated with a scheduler such as Quartz.

Task parallelism – leverage Spring Batch’s parallel processing to accelerate large‑scale data handling.

Error handling – decide whether to skip, abort, or raise alerts based on business scenarios.

Task timeout – define handling logic for RPC‑induced timeouts.

Memory overflow – use bounded queues (e.g., ArrayBlockingQueue ) to keep memory usage under control.

Duplicate execution – implement termination policies and alert mechanisms to avoid endless resource consumption.

Parallel Data Retrieval

For database‑centric workloads, Spring Batch can combine its parallel processing with MyBatis to read and write data efficiently, and even employ read‑write separation to protect online transaction performance.

2 Data Layer

Batch jobs typically ingest data either from files or directly from databases. Reading one record at a time can cause excessive network I/O and latency.

IO Considerations

Increasing parallelism reduces I/O cost. When using an ORM such as MyBatis, batch insert/update capabilities should be employed, but packet size limits and database‑side constraints must be respected.

MyBatis Integration

Batch processing & large packet handling – MyBatis supports bulk operations, yet the application must cap batch sizes to stay within database limits.

Spring Batch integration – MyBatis provides MyBatisPagingItemReader and MyBatisBatchItemWriter , which fit naturally into Spring Batch’s chunk model.

Database Concerns

When batch volume exceeds a single table’s recommended limits, strategies such as sharding (splitting databases/tables) and archiving historical data become essential. Parallel loading from multiple data sources can further improve throughput.

3 Deployment Considerations

Batch modules should be placed close to their data sources to minimize latency. Each business domain is better served by an independent batch module rather than a monolithic shared one, which simplifies data ownership and database management.

Isolation from OLTP

The batch service should be deployed separately from online transaction processing (OLTP) services, with its own instances and, in high‑availability setups, multiple data‑center deployments (e.g., two sites with four instances each for disaster recovery).

4 Monitoring and Management

Effective monitoring includes tracking job statistics (success, failure, termination, running counts) and generating alerts for abnormal conditions. A console or JMX‑based dashboard can provide real‑time visibility.

Configuration Hot‑Reload

Support for dynamic updates of static resources or job metadata without restarting the service is desirable.

Fault Tolerance and Checkpointing

Implementing checkpoint/retry mechanisms and graceful failover ensures that long‑running jobs can resume after interruptions, especially in cloud‑native environments where permission controls may differ.

Reference

https://www.cnblogs.com/jietang/p/5353220.html

https://www.cnblogs.com/javastack/p/15105397.html

https://blog.csdn.net/csucsgoat/article/details/116724221

https://mybatis.org/spring/batch.html

https://www.cnblogs.com/javastack/p/15105397.html