Retrieval-augmented generation (RAG) typically benefits from a larger knowledge base, but a new arXiv paper warns that this is not always true. The study, "When More Documents Hurt RAG: Measuring and Reducing Vector Search Dilution with Domain-Scoped Retrieval," shows that RAG performance degrades when applied to large, heterogeneous document collections.
The problem lies in how dense retrieval works. As the document set grows and becomes more varied, the similarity scores produced by dense encoders become less discriminative. This means the top-k retrieval step returns chunks that are semantically less relevant, even though they score highly. The authors call this effect vector search dilution.
To counter this, they propose domain-scoped retrieval, which restricts the search space to a relevant domain before applying dense retrieval. By narrowing the candidate set, the method reduces dilution and helps maintain the quality of retrieved evidence. The paper offers a concrete way to measure and mitigate a failure mode that becomes more pronounced as RAG systems scale.