发现大模型搜索过度问题，提出新评估指标并给出优化方向
Over-Searching in Search-Augmented Large Language Models
- 通过多维度测试发现模型在复杂任务中易过度调用搜索
- 引入TPC指标衡量搜索效率，揭示噪声检索会加剧过度搜索
- 适合研究高效检索增强型大模型的开发者与研究人员
检索增强的大语言模型在知识密集型任务中表现优异，但常出现过度搜索问题——即使无需外部检索也能获得高质量回答时仍频繁调用搜索工具，导致计算浪费和因引入无关信息引发幻觉。本文系统评估了多种场景下的过度搜索现象，发现：(i) 检索可提升可回答问题的准确率，但会降低不可回答问题的拒答率；(ii) 复杂推理模型与深度研究系统更易过度搜索，噪声检索加剧该问题，且在多轮对话中逐轮累积；(iii) 检索结果中包含否定证据能有效提升拒答能力。为此，我们提出Tokens Per Correctness（TPC）指标，量化搜索性能与成本的权衡。最后，我们在查询与检索层面探索缓解方法，并发布OverSearchQA数据集，推动高效检索增强模型的研究。
原文摘要 · Abstract (English)
Search-augmented large language models (LLMs) excel at knowledge-intensive tasks by integrating external retrieval. However, they often over-search -- unnecessarily invoking search tool even when it does not improve response quality, which leads to computational inefficiency and hallucinations by incorporating irrelevant context. In this work, we conduct a systematic evaluation of over-searching across multiple dimensions, including query types, model categories, retrieval conditions, and multi-turn conversations. Our finding shows: (i) search generally improves answer accuracy on answerable queries but harms abstention on unanswerable ones; (ii) over-searching is more pronounced in complex reasoning models and deep research systems, is exacerbated by noisy retrieval, and compounds across turns in multi-turn conversations; and (iii) the composition of retrieved evidence is crucial, as the presence of negative evidence improves abstention. To quantify over-searching, we introduce Tokens Per Correctness (TPC), an evaluation metric that captures the performance-cost trade-off for search-augmented LLMs. Lastly, we investigate mitigation approaches at both the query and retrieval levels and release the OverSearchQA to foster continued research into efficient search-augmented LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。