How Java + LangChain4j Can Eliminate Messy Chunking for High‑Quality RAG Document Splitting

In Retrieval‑Augmented Generation (RAG) systems, the recall ceiling is often set by the quality of document chunking; fixed‑size chunking can sever critical semantic links, causing the embedding model to miss essential context.

Why Fixed‑Size Chunking Fails

When a crucial logical sentence spans characters 499‑505, a naïve 500‑character split places the first half in Chunk A and the second half in Chunk B. Each fragment is embedded separately, breaking the original meaning and dramatically lowering recall for queries that target either half.

Solution 1: Recursive Punctuation‑Based Splitting

This widely used, cost‑effective method avoids hard character limits. The algorithm attempts a hierarchy of delimiters: first split by double newlines ( \n\n) to separate paragraphs; if a segment remains too long, split by single newlines ( \n); then by periods ( 。); and finally by commas. This respects the natural “breathing rhythm” of language.

Solution 2: Overlap Windows

Even with recursive splitting, long‑text boundaries can still cause context loss. Introducing a 10‑20% overlap (e.g., 50 tokens for a 500‑token chunk) ensures that the start of each chunk repeats the tail of the previous one, preserving continuity. Accurate token counts require a tokenizer that matches the target LLM (e.g., OpenAI’s tokenizer for GPT‑4).

Solution 3: Parent‑Child Document Mapping

To balance recall precision and context richness, store large parent paragraphs in a key‑value store (Redis) and the short child sentences in a vector store (Qdrant) with a parent_id metadata field linking back to the parent. During ingestion, each parent receives a UUID, is saved in Redis, then split into child sentences that are embedded and stored in Qdrant together with the parent_id. During retrieval, the query is embedded, the top‑N child chunks are fetched from Qdrant, their unique parent_id s are collected, and the full parent texts are pulled from Redis for final LLM consumption.

Java/LangChain4j Implementation

import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.document.DocumentSplitter;
import dev.langchain4j.data.document.splitter.DocumentSplitters;
import dev.langchain4j.model.openai.OpenAiTokenizer;

public class DocumentProcessService {
    public List<TextSegment> processWithOverlap(Document document) {
        // 1. Define tokenizer (example: OpenAI; replace with HuggingFace for private deployment)
        Tokenizer tokenizer = new OpenAiTokenizer("gpt-4");

        // 2. Create recursive splitter with overlap
        int maxTokens = 500; // each chunk max 500 tokens
        int overlapTokens = 50; // 10% overlap
        DocumentSplitter splitter = DocumentSplitters.recursive(
                maxTokens,
                overlapTokens,
                tokenizer
        );

        // 3. Perform splitting
        return splitter.split(document);
    }
}

Ingestion code that builds the parent‑child structure:

public void ingestParentChild(String largeText) {
    // 1. Split large text into parent paragraphs (e.g., double newline)
    List<String> parentChunks = splitIntoParagraphs(largeText);

    for (String parentText : parentChunks) {
        String parentId = UUID.randomUUID().toString();

        // Store parent in Redis
        redisTemplate.opsForValue().set("doc:parent:" + parentId, parentText);

        // Split parent into child sentences
        List<String> childChunks = splitIntoSentences(parentText);
        List<TextSegment> childSegments = new ArrayList<>();
        for (String childText : childChunks) {
            Metadata metadata = new Metadata();
            metadata.put("parent_id", parentId);
            childSegments.add(TextSegment.from(childText, metadata));
        }

        // Embed children and store in Qdrant
        embeddingStore.addAll(embeddingModel.embedAll(childSegments).content(), childSegments);
    }
}

Custom retriever that stitches child matches back to their parents:

@Component
@RequiredArgsConstructor
public class ParentChildRetriever implements ContentRetriever {
    private final EmbeddingStore<TextSegment> qdrantStore;
    private final EmbeddingModel embeddingModel;
    private final StringRedisTemplate redisTemplate;

    @Override
    public List<Content> retrieve(Query query) {
        // 1. Embed the user query
        Embedding queryEmbedding = embeddingModel.embed(query.text()).content();

        // 2. Retrieve top‑5 child chunks from Qdrant
        List<EmbeddingMatch<TextSegment>> matches = qdrantStore.findRelevant(queryEmbedding, 5);

        // 3. Collect unique parent IDs (deduplicate possible multiple hits from the same parent)
        Set<String> parentIds = matches.stream()
                .map(match -> match.embedded().metadata().getString("parent_id"))
                .collect(Collectors.toSet());

        // 4. Fetch full parent paragraphs from Redis
        List<Content> finalContents = new ArrayList<>();
        for (String parentId : parentIds) {
            String parentText = redisTemplate.opsForValue().get("doc:parent:" + parentId);
            if (parentText != null) {
                finalContents.add(Content.from(parentText));
            }
        }
        // 5. Return the assembled contents for the LLM
        return finalContents;
    }
}

Metadata Injection for Global Context

After chunking, each slice should retain its source metadata (e.g., document title, chapter, page). For example, a legal excerpt becomes:

[《2023年刑法经典案例》 - 抢劫罪章节 - 第12页] 张三被判处有期徒刑三年。

Embedding this enriched text ensures the vector store captures the full semantic context.

Outcome

Applying the three‑step approach—recursive punctuation splitting, overlapping windows, and parent‑child mapping—significantly improves RAG recall while keeping the LLM input concise and context‑rich. The article emphasizes that true RAG optimization requires meticulous non‑structured data governance.