How to Add Retry Logic to Spring Batch Jobs for Resilient Processing

1. Introduction

By default, a Spring Batch job fails when any error occurs during execution. To improve resilience, we need to handle intermittent failures. This article explains how to configure retry logic in the Spring Batch framework.

If you are unfamiliar with Spring Batch, refer to the introductory tutorial linked earlier.

2. Simple Example

Assume a batch job reads a CSV file as input:

username, userid, transaction_date, transaction_amount
sammy, 1234, 31/10/2015, 10000
john, 9999, 3/12/2015, 12321

The job then calls a REST endpoint for each record to obtain the user's age and postCode :

public class RetryItemProcessor implements ItemProcessor<Transaction, Transaction> {
    @Override
    public Transaction process(Transaction transaction) throws IOException {
        log.info("RetryItemProcessor, attempting to process: {}", transaction);
        HttpResponse response = fetchMoreUserDetails(transaction.getUserId());
        // parse user's age and postCode from response and update transaction
        ...
        return transaction;
    }
    ...
}

Finally, it generates a merged XML output:

<transactionRecord>
    <transactionRecord>
        <amount>10000.0</amount>
        <transactionDate>2015-10-31 00:00:00</transactionDate>
        <userId>1234</userId>
        <username>sammy</username>
        <age>10</age>
        <postCode>430222</postCode>
    </transactionRecord>
    ...
</transactionRecord>

3. Adding Retry to ItemProcessor

If a REST call times out, the batch job would fail. To retry failed items, we configure the job to retry up to three times:

@Bean
public Step retryStep(ItemProcessor<Transaction, Transaction> processor,
                      ItemWriter<Transaction> writer) throws ParseException {
    return stepBuilderFactory
        .get("retryStep")
        .<Transaction, Transaction>chunk(10)
        .reader(itemReader(inputCsv))
        .processor(processor)
        .writer(writer)
        .faultTolerant()
        .retryLimit(3)
        .retry(ConnectTimeoutException.class)
        .retry(DeadlockLoserDataAccessException.class)
        .build();
}

The faultTolerant() method enables retry, and retry together with retryLimit define the exception types and maximum attempts.

4. Testing Retry Attempts

A test scenario simulates the REST endpoint being unavailable for the first two calls and succeeding on the third:

@Test
public void whenEndpointFailsTwicePasses3rdTime_thenSuccess() throws Exception {
    // setup mock responses
    when(httpClient.execute(any()))
        .thenThrow(new ConnectTimeoutException("Timeout count 1"))
        .thenThrow(new ConnectTimeoutException("Timeout count 2"))
        .thenReturn(httpResponse);
    JobExecution jobExecution = jobLauncherTestUtils.launchJob(defaultJobParameters());
    assertThat(jobExecution.getExitStatus().getExitCode(), is("COMPLETED"));
}

The logs show the first record (id=1234) failing twice and succeeding on the third retry.

Another test verifies that when all retries are exhausted, the job fails:

@Test
public void whenEndpointAlwaysFail_thenJobFails() throws Exception {
    when(httpClient.execute(any()))
        .thenThrow(new ConnectTimeoutException("Endpoint is down"));
    JobExecution jobExecution = jobLauncherTestUtils.launchJob(defaultJobParameters());
    assertThat(jobExecution.getExitStatus().getExitCode(), is("FAILED"));
}

5. XML Configuration Equivalent

The same retry configuration can be expressed in XML:

<batch:job id="retryBatchJob">
    <batch:step id="retryStep">
        <batch:tasklet>
            <batch:chunk reader="itemReader" writer="itemWriter"
                         processor="retryItemProcessor" commit-interval="10"
                         retry-limit="3">
                <batch:retryable-exception-classes>
                    <batch:include class="org.apache.http.conn.ConnectTimeoutException"/>
                    <batch:include class="org.springframework.dao.DeadlockLoserDataAccessException"/>
                </batch:retryable-exception-classes>
            </batch:chunk>
        </batch:tasklet>
    </batch:step>
</batch:job>

6. Summary

This article demonstrated how to configure retry logic in Spring Batch using both Java and XML configurations, and how unit tests can be used to observe retry behavior in practice.