Spring AI Day 9: Automating Document ETL to Feed Enterprise Knowledge into a Vector Store

ETL Three Steps: Extract → Transform → Load

Spring AI models the ingestion pipeline as a classic ETL process, mapping each phase to a functional interface: Supplier for reading, Function for transforming, and Consumer for writing.

Extract – Reading Various Document Formats

Use TikaDocumentReader from the spring-ai-tika-document-reader module to read PDFs, Word, PPT, HTML, and dozens of other formats. Add the Maven dependency:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-tika-document-reader</artifactId>
</dependency>

Example code reads a PDF resource and returns a List<Document>:

// Read a PDF product manual
TikaDocumentReader reader = new TikaDocumentReader(new ClassPathResource("/产品手册.pdf"));
List<Document> docs = reader.read();

For plain‑text files, the lighter TextReader is used:

TextReader textReader = new TextReader(new ClassPathResource("/faq.txt"));
List<Document> docs = textReader.read();

Transform – Splitting Long Documents into Chunks

Long documents are split by token count using TokenTextSplitter. A typical chunk size of 800 tokens balances retrieval precision and context retention.

TokenTextSplitter splitter = TokenTextSplitter.builder()
        .withChunkSize(800)   // approx. 800 tokens per chunk
        .build();

List<Document> chunks = splitter.apply(docs);

The chunk size can be tuned per document type.

Load – Writing Embeddings to a Vector Store

The vector store acts as the writer; calling vectorStore.accept(chunks) automatically vectorizes and persists the chunks.

vectorStore.accept(
        splitter.apply(
                new TikaDocumentReader(resource).read()));

Running the pipeline once converts an entire manual into searchable semantic vectors; new documents can be re‑run to append.

Practical example: A 120‑page product PDF is processed, yielding about 200 semantic chunks that are stored in the vector store with a single code snippet and a few seconds of execution, without any manual annotation.

Day 9 Summary

ETL: read → transform → write, standard knowledge‑ingestion pipeline.

TikaDocumentReader: reads PDF/Word/PPT/HTML.

TextReader: reads plain text.

TokenTextSplitter: splits by token count.

vectorStore.accept(): auto‑vectorizes and stores.