用分层聚类自动选相关文档,省去手动调k,提升法律文本检索准确率。
Tree-Based Text Retrieval via Hierarchical Clustering in RAGFrameworks: Application on Taiwanese Regulations
- 通过分层聚类动态选择最相关文档,无需预设k值
- 在台湾法规数据集上专家评估表现更优,精度高且结果可解释
- 方法简单易集成,适合资源有限的实际场景
传统检索增强生成(RAG)系统采用暴力内积搜索获取最相似的top-k文档,再与用户查询一起输入语言模型,以访问外部知识并减少幻觉。然而,在实际应用中选择合适的k值仍是一个重大挑战:k过小可能无法获取足够信息,k过大则引入过多无关内容。为此,我们提出一种基于分层聚类的检索方法,无需预先设定k值。该方法在保持系统响应准确性与相关性的基础上,自适应选择语义相关的文本内容。实验阶段,我们将该方法应用于带有专家评分的台湾法规数据集。结果表明,该方法在专家评估中表现更优,同时保持高精度,且无需预定义k值,展示了在法律文本检索任务中更高的准确性和可解释性。本框架实现简单,易于集成到现有RAG流程中,是一种适用于资源受限环境下的实用解决方案。
原文摘要 · Abstract (English)
Traditional Retrieval-Augmented Generation (RAG) systems employ brute-force inner product search to retrieve the top-k most similar documents, then combined with the user query and passed to a language model. This allows the model to access external knowledge and reduce hallucinations. However, selecting an appropriate k value remains a significant challenge in practical applications: a small k may fail to retrieve sufficient information, while a large k can introduce excessive and irrelevant content. To address this, we propose a hierarchical clustering-based retrieval method that eliminates the need to predefine k. Our approach maintains the accuracy and relevance of system responses while adaptively selecting semantically relevant content. In the experiment stage, we applied our method to a Taiwanese legal dataset with expert-graded queries. The results show that our approach achieves superior performance in expert evaluations and maintains high precision while eliminating the need to predefine k, demonstrating improved accuracy and interpretability in legal text retrieval tasks. Our framework is simple to implement and easily integrates with existing RAG pipelines, making it a practical solution for real-world applications under limited resources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。