Why Fixed-Size Chunking Fails in RAG: Interview Insights

Problem Statement

Fixed‑size chunking cuts a document into equal‑length fragments based solely on character or token count. It ignores semantic boundaries, document structure, and topic cohesion, which leads to incomplete or noisy retrieval results in Retrieval‑Augmented Generation (RAG) pipelines.

Fatal Issues

Semantic break – a sentence, paragraph or logical argument may be split in the middle, so the retrieved chunk contains only partial information. Example: a comparison "why choose Milvus over Pinecone" was divided into three chunks; only one chunk was retrieved, producing a nonsensical answer.

Topic scattering – content about the same topic can be spread across multiple chunks, causing the retriever to hit only a fragment.

Information redundancy or loss – too much overlap wastes storage; too little overlap discards key context.

Lack of structure awareness – headings, lists, and sections are treated uniformly, so structural cues are lost.

Retrieval noise – a chunk may mix unrelated topics, leading to irrelevant matches.

One‑sentence summary : Fixed‑size chunking "looks only at length, not meaning", breaking semantic boundaries.

Overlap as a Patch

Many implementations add a token overlap to mitigate sentence cuts. The following Java snippet shows a typical LangChain4j configuration:

import dev.langchain4j.data.document.DocumentSplitter;
import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.segment.TextSegment;
import dev.langchain4j.data.document.DocumentSplitters;

DocumentSplitter splitter = DocumentSplitters.recursive(
    500,   // max tokens per chunk
    100    // overlap tokens
);
Document document = Document.from(documentText);
List<TextSegment> segments = splitter.split(document);

Overlap introduces new drawbacks:

Storage bloat – a 20 % overlap increases vector‑store size and embedding cost proportionally.

Retrieval redundancy – duplicated content consumes the LLM’s context window.

Partial mitigation – it only reduces sentence cuts; topic scattering remains.

Alternative Chunking Strategies

Recursive character splitting – splits first by paragraph, then sentence, then characters. Balances size control and semantic coherence. Supported by LangChain4j and Spring AI.

Semantic chunking – detects changes in embedding similarity to place boundaries. Provides the best semantic integrity but requires an embedding model for each split and higher compute.

Document‑structure chunking – uses headings, sections, or list markers as split points. Preserves logical hierarchy; requires well‑formed markup (Markdown, HTML).

Agentic chunking – lets an LLM decide where to cut. Yields the most intelligent boundaries but is very costly and slow, suitable for frontier research.

Production‑Grade Practices

Prefer recursive character splitting over pure fixed‑size chunking.

If the source is Markdown or HTML, split by heading hierarchy so each chunk has a clear topic.

Attach metadata (source file name, section title, page number) to every chunk for filtered retrieval.

Choose chunk size based on use case: 200‑500 tokens for question answering, larger for summarisation. A common starting point is 300 tokens with 50‑token overlap, then tune empirically.

Maintain multi‑granularity chunks – coarse chunks for context supplementation and fine chunks for precise matching.

Spring AI example (token‑based splitter with metadata):

import org.springframework.ai.document.Document;
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
import java.util.List;

TokenTextSplitter splitter = new TokenTextSplitter(
    500,   // target token count
    100,   // overlap
    50,    // min chars per chunk
    100,   // min length to embed
    0,     // max number of chunks (0 = unlimited)
    true   // keep separator
);
List<Document> chunks = splitter.apply(originalDocuments);
chunks.forEach(chunk -> {
    chunk.getMetadata().put("source", "technical_doc.pdf");
    chunk.getMetadata().put("section", "Chapter 3‑Architecture Design");
    chunk.getMetadata().put("page", 42);
});

Evaluation Method

To compare strategies, prepare a benchmark set of queries with ground‑truth answers. Measure retrieval recall and answer accuracy. The RAGAS framework can automate this, reporting Context Precision (fraction of relevant content in retrieved chunks) and Context Recall (whether all needed chunks were retrieved).

Frontier Trends (2025‑2026)

Late Chunking (Jina AI) – embed the whole document first, then chunk; each chunk’s vector contains global context, reducing semantic breaks.

Max‑Min semantic chunking (2025 paper) – combines semantic similarity with a Max‑Min algorithm to auto‑detect boundaries, outperforming traditional methods.

Context‑aware chunking – automatically prepends a short summary of surrounding text to each chunk, giving every chunk a built‑in context wrapper.

Multimodal chunking – processes text, tables, code, and formulas separately instead of a single text‑only strategy.