Spring Batch for Massive Data Migration: Partitioning, Fault Tolerance & Checkpoint Restart

1. Why Traditional Scripts Fail at Scale

Teams often start with Crontab + Java/Shell scripts, but as data grows (10M+ orders, GB-sized logs), four fatal flaws emerge:

OOM is routine : SELECT * or File.readAllBytes() loads everything into heap, crashing the JVM before logs flush.

State black box : Mid-run crashes leave no record of processed rows; recovery requires manual log digging.

Hard-coded fault tolerance : Format errors or unique-key violations abort the whole job; naive retries hammer the connection pool.

Scaling is hard : Single-threaded serial execution chokes on 50GB+ files; custom sharding risks skew and duplicates.

Batch processing demands four capabilities: streaming I/O for constant memory, partitioning for throughput, Chunk-defined transaction boundaries, and metadata persistence for resumable runs . Spring Batch aligns exactly with these.

2. Core Model: Job / Step / Chunk

2.1 Layered Responsibilities

Job : Top-level container representing a complete business flow; sequences multiple Steps and defines restart policy.

Step : Unit of work; transaction control, fault tolerance, and listeners attach here.

Chunk : The framework's soul — not a data size but a transaction commit boundary . Configuring chunk(1000) means: read 1000 items → process → write → commit transaction → record progress. This micro-batch commit eliminates long-transaction lock contention and undo-log bloat.

2.2 Data Flow Pipeline

ItemReader : Returns one record per read(); must be stateless or cursor-driven. Built-in readers pull on demand, keeping memory flat.

ItemProcessor : Transforms, filters, validates. Returning null drops the item silently. Keep it lightweight — no heavy compute or blocking I/O.

ItemWriter : Receives a List of chunk size, batch-flushes to target. Writer ignores transactions; Step proxies the commit.

Because only a fixed number of objects live in memory at once, OOM is structurally impossible.

3. Production-Grade Configuration

3.1 Partitioned Parallelism (Partitioner)

For 100M+ rows or 50GB+ files, PartitionStep splits data across workers.

@Bean
public Step partitionedStep(PartitionHandler partitionHandler) {
    return stepBuilderFactory.get("partitionedStep")
        .partitioner("workerStep", rangePartitioner())
        .partitionHandler(partitionHandler)
        .build();
}

@Bean
public Partitioner rangePartitioner() {
    return gridSize -> {
        long minId = 1L;
        long maxId = 10000000L;
        long range = (maxId - minId + 1) / gridSize;
        Map<String, ExecutionContext> map = new HashMap<>();
        for (int i = 0; i < gridSize; i++) {
            ExecutionContext ctx = new ExecutionContext();
            ctx.putLong("startId", minId + i * range);
            ctx.putLong("endId", (i == gridSize - 1) ? maxId : minId + (i + 1) * range - 1);
            map.put("partition_" + i, ctx);
        }
        return map;
    };
}

Pair with TaskExecutorPartitionHandler and a thread pool for intra-node parallelism; for multi-node, serialize ExecutionContext via Spring Cloud Data Flow or K8s Jobs.

3.2 Streaming Large File Reads

FlatFileItemReader

wraps a BufferedReader, reading one line at a time — memory stays flat.

@Bean
public FlatFileItemReader<UserRecord> flatFileItemReader() {
    FlatFileItemReader<UserRecord> reader = new FlatFileItemReader<>();
    reader.setResource(new FileSystemResource("/data/export_2023.csv"));
    reader.setLinesToSkip(1); // skip header
    DefaultLineMapper<UserRecord> lineMapper = new DefaultLineMapper<>();
    lineMapper.setLineTokenizer(new DelimitedLineTokenizer(","));
    BeanWrapperFieldSetMapper<UserRecord> fieldMapper = new BeanWrapperFieldSetMapper<>();
    fieldMapper.setTargetType(UserRecord.class);
    lineMapper.setFieldSetMapper(fieldMapper);
    reader.setLineMapper(lineMapper);
    reader.setStrict(false); // don't throw if file missing, aids idempotent retry
    return reader;
}

Custom LineTokenizer handles encoding quirks or non-standard delimiters. Readers are thread-safe per partition instance.

3.3 JDBC Batch Write Tuning

JdbcBatchItemWriter

bottlenecks at network round-trips and driver internals.

@Bean
public JdbcBatchItemWriter<UserRecord> jdbcBatchWriter(DataSource dataSource) {
    JdbcBatchItemWriter<UserRecord> writer = new JdbcBatchItemWriter<>();
    writer.setDataSource(dataSource);
    writer.setSql("INSERT INTO users (id, name, age, created_at) VALUES (:id, :name, :age, :created_at)");
    writer.setItemSqlParameterSourceProvider(new BeanPropertySqlParameterSourceProvider());
    writer.setAssertUpdates(true); // verify affected rows, prevent silent failures
    return writer;
}

Battle-tested tunings:

Align batch size : Chunk size drives addBatch() calls. Keep chunk at 500~2000; larger values blow driver buffers and trigger frequent GC.

Disable generated-key return : Unless business needs auto-increment IDs back, turn off RETURN_GENERATED_KEYS — yields 30%+ throughput gain on MySQL.

Pool & URL params : HikariCP maximumPoolSize must match partition thread count. MySQL JDBC URL requires rewriteBatchedStatements=true; otherwise driver splits batches into single statements.

4. Fault Tolerance & Checkpoint Restart

4.1 Skip vs Retry Configuration

Retry : Transient faults (network blips, DB lock waits, downstream timeouts). Use exponential backoff, not tight loops.

Skip : Poison pills (parse errors, unique violations). Log or dead-letter, then continue.

@Bean
public Step faultTolerantStep(ItemReader<User> reader, ItemWriter<User> writer, PlatformTransactionManager tm) {
    return stepBuilderFactory.get("faultTolerantStep")
        .<User, User>chunk(1000, tm)
        .reader(reader)
        .writer(writer)
        .faultTolerant()
        .retryLimit(3)
        .retry(DeadlockLoserDataAccessException.class, ResourceAccessException.class)
        .skipLimit(50)
        .skip(DataIntegrityViolationException.class, IllegalArgumentException.class)
        .listener(new CustomSkipListener()) // persist skipped records to dead-letter table
        .build();
}

skipLimit

and retryLimit coexist: framework retries first, then skips. Overly broad skip policies leak bad data downstream, breaking reconciliation.

4.2 Transaction Boundaries & Pitfalls

Step fully proxies transactions. All Reader/Writer ops in a Chunk share one transaction; on error the whole Chunk rolls back and metadata stays unchanged — atomicity guaranteed.

Hard-won lesson : Never annotate Processor with @Transactional or open a new transaction via DataSource. This breaks the proxy chain, causing JobRepository progress updates to fail and checkpoint restart to break. Processor must stay pure computation/light validation.

4.3 Correct Checkpoint Restart

Spring Batch stores execution state in six BATCH_* tables. On restart, it reads the last successful Chunk offset and continues.

Critical detail : Restart requires identical JobParameters . JobInstance identity = JobName + JobParameters. Adding a timestamp each launch creates a new instance, disabling restart.

JobParameters params = new JobParametersBuilder()
    .addLong("triggerTime", System.currentTimeMillis()) // ❌ new instance every run
    .addString("source", "order_center")
    .toJobParameters();

To resume, either replay the exact same parameters or use JobExplorer to find the failed JobExecution and call jobLauncher.restart(jobExecution). If new instances are mandatory but manual rollback is needed, enforce idempotency at the business table level ( INSERT IGNORE or ON DUPLICATE KEY UPDATE) rather than fighting framework state.

5. Production Tuning & Observability

5.1 Memory & Cursor Choice

Paging ( JpaPagingItemReader) degrades with deep offsets ( LIMIT offset, size) and risks gaps/duplicates if sort key isn't unique. Cursor ( JdbcCursorItemReader) holds a DB connection for the entire Step duration, easily exhausting the pool.

Guideline: ≤10M rows — use paging with unique sort key (PK). Larger migrations — prefer cursor with FetchSize tuning (MySQL JDBC defaults to Integer.MIN_VALUE for streaming; don't change it) and dedicate a separate connection pool for the reader.

5.2 Long Transactions & Dynamic Chunk Sizing

Don't hard-code chunk size. Peak hours: smaller chunks (500) reduce lock hold time; off-peak: larger (2000) cut commit overhead. Adjust via StepListenerSupport driven by Prometheus metrics, or run two parameterized Steps at different times.

For archive tables with many non-unique indexes, temporarily ALTER TABLE ... DISABLE KEYS, bulk load, then ENABLE KEYS — index rebuild is far faster than per-row maintenance. Never do this on core transactional tables.

5.3 Monitoring Dashboard Integration

Spring Boot Actuator exposes Micrometer metrics; wire to Grafana.

management:
  endpoints:
    web:
      exposure:
        include: health,metrics
  metrics:
    tags:
      application: data-sync-service

Key metrics: spring.batch.job.execution.count — trigger frequency spring.batch.step.read.count / write.count — read/write alignment, detect data loss spring.batch.step.skip.count — spike signals upstream data issues

Custom timer on Processor latency — 80% of bottlenecks live in heavy transformation logic

Alert on skip rate >5% or Job status = FAILED; push to DingTalk/WeCom immediately — don't wait for reconciliation to discover gaps.

6. Selection Guidance & Scheduler Collaboration

Spring Batch isn't universal. Ideal for nightly ETL, cross-DB sync, large-file cleansing — offline jobs where transaction control and state machine shine. For sub-second micro-batches or sliding-window streaming, its metadata persistence and Chunk overhead add unacceptable latency; use Kafka Streams or Flink instead. Lightweight tasks fit simple cron + message queue.

Production standard: decouple scheduling layer from execution layer. Scheduler (XXL-JOB, Airflow, DolphinScheduler) owns dependency orchestration, shard parameter distribution, global retry, alerting. Execution layer (Spring Batch) receives shardIndex / shardTotal, uses Partitioner to slice data ranges, runs streaming processing, manages Chunk transactions and JobRepository state. Scheduler tracks macro progress; Batch governs micro data flow. They sync via webhook or MQ. This combo delivers "schedulable, controllable, resumable, traceable" data pipelines.

Framework is just a tool; stability hinges on anticipating transaction boundaries, data flow directions, and failure scenarios. Tune Chunk granularity, nail Skip policies, cleanly separate scheduling from execution — batch jobs stop being time bombs.