实测发现，大模型用外部知识检索效果有限，小模型反而受益更多。
RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation
- 用混合知识库测试大模型检索增强效果
- 小模型获益显著，大模型提升不明显
- 现有检索策略难应对多样数据源，适合真实场景研究者参考
检索增强生成（RAG）通过在推理时引入外部知识来提升大语言模型（LLM）性能。尽管RAG在基于维基百科等通用语料的基准上表现良好，其在真实、多样检索场景下的有效性仍缺乏深入探索。我们使用MassiveDS——一个包含混合知识的大规模数据存储——评估了RAG系统，发现关键局限：检索主要惠及小模型，重排序器增益微弱，且无单一检索源始终领先。此外，当前大模型难以有效在异构知识源间路由查询。这些发现表明，在部署RAG至真实场景前，亟需开发自适应检索策略。代码与数据见https://github.com/ritaranx/RAG_in_the_Wild。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) enhances large language models (LLMs) by integrating external knowledge retrieved at inference time. While RAG demonstrates strong performance on benchmarks largely derived from general-domain corpora like Wikipedia, its effectiveness under realistic, diverse retrieval scenarios remains underexplored. We evaluated RAG systems using MassiveDS, a large-scale datastore with mixture of knowledge, and identified critical limitations: retrieval mainly benefits smaller models, rerankers add minimal value, and no single retrieval source consistently excels. Moreover, current LLMs struggle to route queries across heterogeneous knowledge sources. These findings highlight the need for adaptive retrieval strategies before deploying RAG in real-world settings. Our code and data can be found at https://github.com/ritaranx/RAG_in_the_Wild.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。