通过动态聚类压缩文档，减少检索冗余，提升大模型生成准确性
Efficient Dynamic Clustering-Based Document Compression for Retrieval-Augmented-Generation
- 基于潜在文档关系动态聚类，自动过滤无关内容
- 在多个问答与幻觉检测数据集上表现更优，效果稳定
- 适合需要高精度知识注入的RAG应用，如智能客服、学术助手
检索增强生成（RAG）近年来成为大语言模型推理中注入知识的主流方法。然而，现有RAG系统因难以利用细粒度文档间关系，常面临检索噪声和冗余内容问题，影响生成结果的准确性。为此，本文提出高效动态聚类文档压缩框架EDC2-RAG，通过挖掘文档间的潜在关联，在去除无关信息和冗余内容的同时保留关键知识。我们在GPT-3.5-Turbo和GPT-4o-mini基础上，于多个通用知识问答与幻觉检测数据集上验证该方法。实验结果表明，该方法在不同场景与设置下均实现一致性能提升，展现出强鲁棒性与广泛适用性。代码与数据集已开源：https://github.com/Tsinghua-dhy/EDC-2-RAG。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has emerged as a widely adopted approach for knowledge injection during large language model (LLM) inference in recent years. However, due to their limited ability to exploit fine-grained inter-document relationships, current RAG implementations face challenges in effectively addressing the retrieved noise and redundancy content, which may cause error in the generation results. To address these limitations, we propose an Efficient Dynamic Clustering-based document Compression framework (EDC2-RAG) that utilizes latent inter-document relationships while simultaneously removing irrelevant information and redundant content. We validate our approach, built upon GPT-3.5-Turbo and GPT-4o-mini, on widely used knowledge-QA and Hallucination-Detection datasets. Experimental results show that our method achieves consistent performance improvements across various scenarios and experimental settings, demonstrating strong robustness and applicability. Our code and datasets are available at https://github.com/Tsinghua-dhy/EDC-2-RAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。