RAG Data Quality: Old Problems in a New Bottle

Redundancy and Approximate Duplicates

Legal statements and templated clauses that are stored verbatim create dense clusters of similar vectors in the embedding space. Under approximate nearest‑neighbor (ANN) search, these clusters consume retrieval slots, raising the proportion of noise in top‑k results. Detecting such issues requires more than exact‑match checks; near‑duplicate detection algorithms like MinHash or SimHash are needed, and removal must distinguish truly redundant content from superficially similar but semantically distinct versions.

Why More Documents Do Not Guarantee Better Results

After noisy data enters the retrieval stage, three generation‑stage problems arise:

Position sensitivity : Large language models (LLMs) may forget relevant information that appears in the middle of long contexts, a phenomenon known as “lost in the middle”.

Attention dilution : Irrelevant chunks increase the overall “noisiness” of the context, scattering the model’s attention away from truly pertinent content.

Retrieval quantity paradox : Adding more retrieved chunks improves recall but also injects additional noise, making it harder for the model to discern relevant facts.

Consequently, retrieving more documents is not inherently beneficial; the model must filter out irrelevant or misleading chunks before generation.

Explicit Chunk‑Level Filtering

Instead of coarse document‑level relevance judgments, a finer‑grained filter evaluates each chunk against the user query, assessing relevance at the fragment level. Traditional methods that improve overall document relevance still allow unrelated chunks to reach the generation stage.

Privacy and Compliance Risks

Many teams mistakenly believe that once text is encoded into floating‑point embeddings, it becomes an irreversible, hash‑like representation. In reality, embeddings are lossy compressions without a simple mathematical inverse, yet they can be approximated through embedding inversion attacks when an attacker knows the model and the original text is short.

Mitigations include encrypting stored vectors (potentially with homomorphic encryption despite performance costs), restricting access to the embedding model, and auditing stored metadata because the real leakage often lies in accompanying raw text or identifiers rather than the vectors themselves.

Continuous Governance

One‑off cleaning only solves the current batch of data; production systems continuously ingest new documents, generating fresh dirty data. Effective governance must therefore be an ongoing process measured by retrieval‑quality metrics such as recall and mean reciprocal rank (MRR) before and after cleaning. Over‑cleaning can also introduce new problems by deleting critical information, so end‑to‑end downstream performance must be used to validate any data‑quality action.

Conclusion

The capability of the LLM sets the lower bound of a RAG system, but data quality determines its upper bound. No amount of sophisticated retrieval or powerful generation can fully compensate for fundamental data‑layer defects, making long‑term, iterative data‑quality engineering essential.