评测AI搜索信息的完整性,发现现有模型仅能找回不到一半内容。
SeekerGym: A Benchmark for Reliable Information Seeking

- 用维基百科和机器学习综述论文构建搜索任务,评估信息完整度
- 顶尖模型在维基百科上仅召回42.5%段落,在综述论文中为29.2%
- 强调模型应量化信息缺失程度,适合研究可信AI代理的学者参考
尽管人工智能代理在诸多任务中表现优异,但在可信性方面仍面临根本挑战。以深度研究代理为例,其虽能有效检索信息,但无法保证结果的完整性,信息遗漏可能引入偏差,即使已提供内容正确且相关。为此,我们提出SeekerGym,一个用于评估AI代理信息检索完整性的基准。该基准还衡量代理对信息不完整性的不确定性量化能力:若未能获取全部相关信息,至少应能估计缺失比例。每个任务对应一篇文档(如维基百科文章),代理需通过查询获取其中段落;因文档全面覆盖主题,成功检索部分可直接反映检索完整性。除维基百科外,还涵盖机器学习综述论文,目标是提取相关章节。我们在多个模型与算法上进行测试,最佳方法在维基百科上仅检索到42.5%的段落,在机器学习综述论文中为29.2%,仍有巨大提升空间。
原文摘要 · Abstract (English)
Despite their substantial successes, AI agents continue to face fundamental challenges in terms of trustworthiness. Consider deep research agents, tasked with searching for information relevant to a given topic-while AI agents can perform effective information retrieval, there is little guarantee regarding the completeness of this information. Gaps in retrieved information can leave biases that mislead users even if the information they are given is correct and relevant. We introduce SeekerGym, a benchmark designed to evaluate the completeness of information retrieved by AI agents. In addition, SeekerGym also measures how well agents quantify their uncertainty in the completeness of their information; if an agent fails to retrieve all relevant information, it is useful for it to at least quantify how much might be missing. At a high level, each task in SeekerGym is a document (e.g., a Wikipedia article), and the AI agent must issue queries to retrieve passages from that document. Intuitively, the document comprehensively covers a topic, so the ability to retrieve its sections directly measures completeness of information retrieval. In addition to Wikipedia, we also consider machine learning survey papers, where the goal is to retrieve relevant sections of a survey paper. We benchmark several models and algorithms; the best approaches retrieve 42.5% of passages on Wikipedia and 29.2% on ML Surveys, leaving substantial room for improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。