首个针对查询错误的RAG鲁棒性评测基准,提升大模型在实际使用中的准确率。
QE-RAG: A Robust Retrieval-Augmented Generation Benchmark for Query Entry Errors
- 构建含20%和40%输入错误的六大数据集,模拟真实用户打字错误。
- 发现现有RAG方法在查询错误下性能显著下降,最高降幅达35%。
- 提出对比学习与检索增强纠错方法,兼容主流RAG框架并提升鲁棒性。
检索增强生成(RAG)已成为提升大语言模型事实准确性的重要方法。现有基准普遍假设用户查询无误,但在真实交互中,键盘邻近、视觉相似和拼写错误频繁发生。本文提出首个专为查询输入错误设计的鲁棒性评测基准QE-RAG,通过在六个常用数据集上以20%和40%的比例注入三类常见输入错误,模拟真实场景。实验表明,错误查询使模型性能下降,可通过查询修正和训练鲁棒检索器缓解。基于此,我们提出一种基于对比学习的鲁棒检索器训练方法和检索增强型查询纠错机制。跨域与同域实验验证:(1)当前先进RAG方法(包括序列、分支、迭代型)对查询错误敏感,性能下降显著;(2)所提方法能有效提升鲁棒性,且可无缝集成至现有RAG系统,进一步改善其表现。
原文摘要 · Abstract (English)
Retriever-augmented generation (RAG) has become a widely adopted approach for enhancing the factual accuracy of large language models (LLMs). While current benchmarks evaluate the performance of RAG methods from various perspectives, they share a common assumption that user queries used for retrieval are error-free. However, in real-world interactions between users and LLMs, query entry errors such as keyboard proximity errors, visual similarity errors, and spelling errors are frequent. The impact of these errors on current RAG methods against such errors remains largely unexplored. To bridge this gap, we propose QE-RAG, the first robust RAG benchmark designed specifically to evaluate performance against query entry errors. We augment six widely used datasets by injecting three common types of query entry errors into randomly selected user queries at rates of 20\% and 40\%, simulating typical user behavior in real-world scenarios. We analyze the impact of these errors on LLM outputs and find that corrupted queries degrade model performance, which can be mitigated through query correction and training a robust retriever for retrieving relevant documents. Based on these insights, we propose a contrastive learning-based robust retriever training method and a retrieval-augmented query correction method. Extensive in-domain and cross-domain experiments reveal that: (1) state-of-the-art RAG methods including sequential, branching, and iterative methods, exhibit poor robustness to query entry errors; (2) our method significantly enhances the robustness of RAG when handling query entry errors and it's compatible with existing RAG methods, further improving their robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。