测试大模型在检索增强下的鲁棒性，发现多数模型表现稳定但仍有提升空间。
Evaluating the Retrieval Robustness of Large Language Models
- 构建1500个开放域问题与维基百科检索结果，评估三种鲁棒性指标。
- 11个大模型在不同检索数量和顺序下仍保持高准确率，平均下降不足10%。
- 适合关注RAG实际应用效果的研究者和开发者参考。
检索增强生成（RAG）通常能提升大语言模型（LLMs）解决知识密集型任务的能力，但因检索不完善及模型利用检索内容能力有限，也可能导致性能下降。本文评估了实际RAG设置中LLMs的鲁棒性（即检索鲁棒性），聚焦三个问题：(1) RAG是否总是优于非RAG；(2) 更多检索文档是否带来更好效果；(3) 文档顺序是否影响结果。为此，我们构建了一个包含1500个开放域问题的基准数据集，每个问题配有来自Wikipedia的检索文档。提出三项鲁棒性度量，分别对应上述问题。涵盖11个LLMs和3种提示策略的全面实验表明，所有模型均表现出意外的高检索鲁棒性；然而，不同程度的不完美鲁棒性仍限制其充分获益于RAG。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) generally enhances large language models' (LLMs) ability to solve knowledge-intensive tasks. But RAG may also lead to performance degradation due to imperfect retrieval and the model's limited ability to leverage retrieved content. In this work, we evaluate the robustness of LLMs in practical RAG setups (henceforth retrieval robustness). We focus on three research questions: (1) whether RAG is always better than non-RAG; (2) whether more retrieved documents always lead to better performance; (3) and whether document orders impact results. To facilitate this study, we establish a benchmark of 1500 open-domain questions, each with retrieved documents from Wikipedia. We introduce three robustness metrics, each corresponds to one research question. Our comprehensive experiments, involving 11 LLMs and 3 prompting strategies, reveal that all of these LLMs exhibit surprisingly high retrieval robustness; nonetheless, different degrees of imperfect robustness hinders them from fully utilizing the benefits of RAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。