How to Efficiently Batch Insert Data with Spring Boot and Elasticsearch

Environment

Spring Boot 2.3.12.RELEASE + Elasticsearch 7.8.0

1. Add Maven Dependencies

<parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>2.3.12.RELEASE</version>
</parent>

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>

2. Configuration

spring.elasticsearch.rest.uris=http://x.x.x.x:9200
# Enable Elasticsearch Repositories (default true)
spring.data.elasticsearch.repositories.enabled=false

3. JPA and Elasticsearch Repository Configuration

When using both Spring Data JPA and Spring Data Elasticsearch, configure their basePackages separately to avoid startup errors. JPA uses JpaRepository, while Elasticsearch uses ElasticsearchRepository.

@EnableElasticsearchRepositories(basePackages = "com.pack.es")
@EnableJpaRepository(basePackages = "com.pack.repository")
public class ESApplication {
    // application entry point
}

If the JPA entity and Elasticsearch document are the same class, rename the Elasticsearch repository (e.g., XXXEsRepository) to avoid naming conflicts.

4. Batch Insertion

@Service
public class TaskDetailEsService {
    @Resource
    private ElasticsearchRestTemplate esTemplate;
    @Resource
    private TaskResultDetailEsRepository esRepository;
    private String index_name = "product";
    BulkOptions options = BulkOptions.builder()
        .withTimeout(TimeValue.timeValueMinutes(2))
        .build();

    public void batchSave(List<Product> datas) {
        long start = System.currentTimeMillis();
        List<IndexQuery> queries = new ArrayList<>();
        int count = 0;
        for (Product detail : datas) {
            IndexQuery indexQuery = new IndexQuery();
            indexQuery.setId(detail.getId());
            indexQuery.setObject(detail);
            queries.add(indexQuery);
            if (count % 1000 == 0) {
                esTemplate.bulkIndex(queries, options, IndexCoordinates.of(index_name));
                queries.clear();
            }
            count++;
        }
        if (!queries.isEmpty()) {
            esTemplate.bulkIndex(queries, options, IndexCoordinates.of(index_name));
            queries.clear();
        }
        esTemplate.indexOps(IndexCoordinates.of(index_name)).refresh();
        System.out.println("Batch save time: " + (System.currentTimeMillis() - start) + " ms");
    }

    public void save(List<TaskResultDetail> datas) {
        long start = System.currentTimeMillis();
        esRepository.saveAll(datas);
        System.out.println("Repository save time: " + (System.currentTimeMillis() - start) + " ms");
    }
}

Both batch insertion approaches show similar performance when inserting 10,000 records.

5. Test Results

Batch save (using ElasticsearchRestTemplate) took 12,916 ms .

Repository save (using saveAll) took 11,916 ms .

The Elasticsearch instance used in the test is a remote service.

End of tutorial.