通过实体过滤减少检索增强生成中的冗余，提升效率。
Reducing Redundancy in Retrieval-Augmented Generation through Chunk Filtering
- 基于实体的轻量级过滤策略，有效去除重复内容。
- 索引规模缩小25%至36%，检索质量接近基线水平。
- 适合追求高效RAG系统部署的研究者与工程师。
标准的检索增强生成（RAG）分块方法常引入大量冗余，增加存储成本并降低检索速度。本研究探索了语义、主题和命名实体等分块过滤策略，旨在在保持检索质量的同时减少索引语料规模。实验在多个语料库上进行，采用基于精确率、召回率和交并比的令牌级评估框架。结果表明，基于实体的过滤可使向量索引规模减少约25%至36%，同时保持接近基线的检索性能。这说明分块过程中引入的冗余可通过轻量级过滤有效消除，显著提升RAG流水线中检索组件的效率。
原文摘要 · Abstract (English)
Standard Retrieval-Augmented Generation (RAG) chunking methods often create excessive redundancy, increasing storage costs and slowing retrieval. This study explores chunk filtering strategies, such as semantic, topic-based, and named-entity-based methods in order to reduce the indexed corpus while preserving retrieval quality. Experiments are conducted on multiple corpora. Retrieval performance is evaluated using a token-based framework based on precision, recall, and intersection-over-union metrics. Results indicate that entity-based filtering can reduce vector index size by approximately 25% to 36% while maintaining high retrieval quality close to the baseline. These findings suggest that redundancy introduced during chunking can be effectively reduced through lightweight filtering, improving the efficiency of retrieval-oriented components in RAG pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。