RAG Chunking Strategies and How to Balance Chunk Size (Full Answer)

Introduction

Building on previous discussions of RAG, fine‑tuning, and long‑context selection, this article provides a systematic answer framework for the interview question "What are the chunking strategies for RAG and how to balance chunk size?" drawing on official documentation and experimental data from Anthropic, LlamaIndex, Pinecone, and Chroma.

Four Common Chunking Strategies

1. Fixed‑Length Splitting

Split text by a fixed number of characters or tokens using a sliding window with overlap (typically 10%–20%). Baseline configuration is around 512 tokens. Pros: simple, fast, cheap. Cons: ignores semantic boundaries, may cut sentences in half.

2. Recursive Splitting

Recursively split by hierarchy of delimiters—paragraphs, then sentences, then words—until each chunk fits a size limit, preserving semantic completeness. Implemented in LangChain’s RecursiveCharacterTextSplitter (default delimiters: "\n\n", "\n", " ", ""). Chroma’s docs recommend overlapping recursive splitting as a solid default.

3. Semantic Splitting

Compute embeddings for adjacent sentences, detect the point where similarity drops fastest, and split there, assuming a topic shift. This yields more semantically cohesive chunks but requires an embedding pass for every sentence, increasing cost.

4. Structure‑Based Splitting

Leverage inherent document structure (e.g., Markdown headings, manual sections) to split by logical units. This preserves context and allows each chunk to retain its position metadata (chapter, section), improving retrieval precision and traceability.

Balancing Chunk Size

The core dilemma is that smaller chunks improve retrieval precision but may lose necessary context, while larger chunks retain context but dilute semantic focus, reducing retrieval accuracy. No universal optimal size exists; experiments on specific corpora are needed to determine the best token count.

Advanced Strategies to Mitigate Context Loss

1. Contextual Retrieval (Anthropic)

Each chunk is prefixed with a short LLM‑generated description of its source document and background, then indexed with both embedding and BM25. Anthropic reports that adding contextual embeddings reduces top‑20 retrieval failure by 35%, adding BM25 lowers it further by 49%, and re‑ranking cuts error from 5.7% to 1.9%. The approach incurs per‑chunk LLM processing cost, mitigated by prompt caching to keep the full text in memory.

2. Late Chunking (Jina AI)

Instead of embedding each pre‑split chunk independently, the entire document (or a macro‑chunk that fits an 8K context window) is passed through a long‑context embedding model (e.g., jina‑embeddings‑v2) to obtain token‑level embeddings. Token embeddings are then mean‑pooled over the original chunk boundaries, producing vectors that incorporate broader context. This requires a model with sufficient context length; for longer texts, macro‑chunks with overlap are created before applying Late Chunking.

Conclusion

When answering interview questions, explain the four basic strategies, the trade‑off of chunk size, and the two advanced methods—Contextual Retrieval (LLM‑generated context) and Late Chunking (long‑context embeddings)—as the top‑level solutions.