测试大模型内部搜索能力,发现检索能提准但易误判,且常因提问和选源出错。
Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs
- 用时间锚点问题测试模型是否因不确定才调用检索
- 动态问题中检索准确率不足70%,主因是提问和选源错误
- 检索后信心虚高,适合关注可信度与检索优化的研究者
现代大模型越来越多地集成内部网络检索以提供实时答案,但其识别信息需求、触发检索及利用检索证据的效能尚不明确。为评估这些机制,我们在不访问模型参数或配置的前提下,通过外部视角评估闭源大模型的检索必要性与有效性。实验设计包括783个时间锚定的事实类查询(可基于截止前知识回答),用于检验检索是否因事实不确定性而触发;以及288个截止后需最新信息的动态查询,用于评估在不可避免的信息需求下检索的有效性。在四个不同规模的模型(两家族)上进行测试。结果显示,启用检索后静态任务准确率显著提升,但置信度校准系统性下降;动态任务中模型虽频繁调用检索,准确率仍低于70%,失败主要源于查询构建与来源选择错误,而非内容整合问题。尽管检索成本低且可选择性使用,但早期失败难以通过重复尝试修复,一旦启用检索,置信度便被过度抬升。总体而言,内部网络检索仅适合作为低延迟验证机制,难以作为可靠的信息检索管道,凸显了在网联大模型中改进检索触发、查询生成与证据感知置信度校准的必要性。
原文摘要 · Abstract (English)
Modern large language models increasingly integrate internal web-based retrieval to provide real-time answers, yet it remains unclear how effectively these systems identify information need, trigger retrieval, and use retrieved evidence. To understand these parameters better, we evaluate the necessity and effectiveness of internal web search through an external lens in closed-source LLMs, without access to model parameters or internal configuration. Our evaluation method comprises a static split of 783 temporally anchored factual queries answerable from pre-cutoff knowledge, designed to test whether retrieval is invoked based on factual uncertainty, and a dynamic split of 288 post-cutoff queries that require up-to-date information, designed to evaluate retrieval effectiveness under unavoidable information need. We experiment across four models across two model families and scales. Across models, enabling retrieval yields substantial accuracy gains on the static split, but systematically degrades confidence calibration. On the dynamic split, models frequently invoke retrieval yet remain below 70 percent accuracy, with failures dominated by query formulation and source selection errors rather than integration of retrieved content. While retrieval is inexpensive to invoke and can be selectively beneficial, repeated retrieval attempts rarely recover from early failures, and confidence becomes inflated once retrieval is available. Overall, internal web-based retrieval functions effectively as a low-latency verification mechanism, but falls short as a reliable IR pipeline, highlighting the need for improved retrieval triggering, query formulation, and evidence-aware confidence calibration in web-enabled LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。