Why Adding More Documents Can Degrade RAG Answers

The recall stage ensures that relevant material is found, while the context assembly stage determines what the model actually sees. Many teams optimize retrieval but then cram dozens of overlapping or version‑conflicting chunks into the prompt, leading to higher token usage, slower responses, and answers that miss conditions, cite outdated rules, or mix rules from different products.

Context windows set a capacity limit, not a guarantee of effective attention.

1. Why More Context Can Harm Answers

Example query: "How to disable auto‑renewal in Enterprise 4.2?" Retrieval returns seven passages, only the first directly answers the question. The others introduce several noise types:

Topic noise: same domain but irrelevant to the current question.

Version conflict: old and new rules appear together.

Entity conflict: personal, enterprise, and overseas editions mixed.

Duplicate information: repeated sentences across adjacent chunks.

Instruction interference: text that looks like a prompt.

Positional issue: key evidence buried in the middle of a long context.

The model must not only locate the correct sentence but also judge which conditions apply and which are obsolete, a task made harder by noisy context.

2. Lost in the Middle Effect

"Lost in the Middle" describes the tendency of LLMs to attend more reliably to the beginning and end of long inputs, while middle information receives less attention, especially when the context is noisy. This positional bias means that even retrieved evidence can be ignored if surrounded by irrelevant material.

3. Contextual Compression Is Not Simple Summarization

Contextual compression aims to keep only evidence needed to answer the current question while discarding irrelevant and duplicate content. Four levels are described:

1. Document‑level Filtering

Decide whether an entire chunk should enter the context. Reranking followed by Top‑N selection belongs here. Fast and structurally stable, but a chunk may contain only a few useful sentences.

2. Sentence‑level Extraction

Extract sentences directly relevant to the query, preserving their source. Example:

Original Chunk: 600 characters with feature description, scope, steps, and notes
Compressed: keep "Applicable to Enterprise 4.2" and three shutdown steps, 120 characters

Extraction retains original wording, which is essential for auditability.

3. Query‑focused Summarization

Generate a summary around the current question, dramatically reducing context length but risking loss of constraints or introducing rewriting bias. For high‑risk tasks, retain document_id, page number, paragraph, and the original excerpt alongside the summary.

4. Structured Compression

For tables, contracts, or API docs, extract only the relevant fields, rows, or sections instead of converting the whole structure into prose. Example: keep table header, target model row, and units when answering a power‑spec question.

4. A Practical Context Assembly Pipeline

Recommended order:

Multi‑retrieval
↓
Permission & Version Filtering
↓
Rerank
↓
Deduplicate & Merge Adjacent Chunks
↓
Sentence Extraction or Query‑focused Summarization
↓
Conflict Detection & Order Adjustment
↓
Assemble Within Token Budget

Step 1: Merge Adjacent Chunks

If top results come from neighboring offsets in the same document, recombine them into a fuller paragraph to avoid duplicate headings and half‑sentences.

def merge_adjacent(chunks):
    chunks = sorted(chunks, key=lambda x: (x.metadata["document_id"], x.metadata["start_offset"]))
    merged = []
    for chunk in chunks:
        if not merged:
            merged.append(chunk)
            continue
        previous = merged[-1]
        same_doc = previous.metadata["document_id"] == chunk.metadata["document_id"]
        close_enough = chunk.metadata["start_offset"] <= previous.metadata["end_offset"] + 100
        if same_doc and close_enough:
            previous.page_content += "
" + chunk.page_content
            previous.metadata["end_offset"] = chunk.metadata["end_offset"]
        else:
            merged.append(chunk)
    return merged

In production, avoid mutating cached documents; copy before merging.

Step 2: Extract Evidence Instead of Re‑writing Answers

Prompt the compression model to copy sentences that support the answer, preserving conditions, negations, versions, units, and exceptions. Example output format:

{
  "chunk_id": "billing-guide-4.2#7",
  "evidence": [
    "Enterprise 4.2 administrators can disable auto‑renewal in billing settings.",
    "The operation must be completed 24 hours before the next billing cycle starts."
  ]
}

Step 3: Assemble Within Token Budget

Reserve space for system prompts, conversation history, user query, and the answer itself. Simple selector example:

def select_with_budget(items, max_tokens, count_tokens):
    selected = []
    used = 0
    for item in items:
        size = count_tokens(item["text"])
        if used + size > max_tokens:
            continue
        selected.append(item)
        used += size
    return selected

Additional heuristics can enforce source diversity and per‑document limits.

5. Ordering Evidence

Three guiding principles:

Strongest evidence first: place the most direct, version‑matched, complete evidence at the top.

Put conflicting content together: if both old and new rules must be kept, keep them adjacent and clearly label their applicable versions.

Avoid burying key evidence: for critical rules, prepend a concise evidence block, followed by the original snippet and source.

Example format:

[Evidence 1]
Source: Enterprise Billing Manual
Version: 4.2
Status: Current
Content: …

[Evidence 2]
Source: Auto‑Renew FAQ
Version: 2026‑06
Status: Current
Content: …

6. Risks of Over‑compression

Compression can drop essential details such as negations, numeric units, version constraints, table headers, step order, and source citations. Therefore evaluation must consider both token reduction and "evidence fidelity".

Metrics to track:

Context token count

Context Precision and Recall

Answer accuracy, Faithfulness, and citation correctness

Compression‑induced latency and cost

For instance, reducing context from 8000 tokens to 1500 tokens while losing a validity‑date condition does not constitute a successful optimization.

7. When Not to Use LLM Compression

Candidate set is already short and accurate.

Regulatory or contract text requires verbatim quoting.

Tabular structure is more important than natural‑language description.

Latency budget is extremely tight.

Compression model cannot be deployed on‑premise for compliance.

Recall and permission problems are still unresolved.

In these cases, prefer deterministic methods: metadata filtering, reranking, adjacent‑chunk merging, title‑based slicing, regex extraction, or row‑level table filtering.

8. Validate with Context A/B Experiments

Run three parallel pipelines on a shared set of hard questions: raw Top‑K, deterministic deduplication only, and deduplication plus LLM compression. Keep the generation model and prompt constant. Measure:

Whether evidence remains complete (negations, conditions, versions, units, source locations).

Whether answer quality truly improves (accuracy, Faithfulness, citation correctness).

Resource trade‑off (token usage, added latency, per‑request cost).

Which question types should bypass compression.

Only adopt compression into the default chain when quality gains persist across evaluation sets and outweigh the added latency and cost.

Conclusion

The goal of RAG is not to feed the model more text but to provide less, more precise, and complete evidence. Top‑K controls candidate size, rerank adjusts priority, and context compression removes the remaining noise, enabling the model to generate accurate, faithful answers.