测试搜索智能体在模糊查询下主动澄清的能力，发现多数模型严重失效。
InteractComp: Evaluating Search Agents With Ambiguous Queries
- 构建9个领域210个带可控歧义的问题，需交互才能解决。
- 最优模型仅13.73%准确率，远低于有完整上下文时的71.50%。
- 揭示模型缺乏交互能力，适合评估和训练智能体的对话纠错能力。
语言智能体在网页搜索与信息检索中展现出巨大潜力，但现有基准大多假设用户查询是完整且无歧义的。这一假设忽略了实际中常见的模糊请求——目标无法在不澄清的情况下确定。然而，大多数智能体缺乏搜索过程中的交互机制，现有评测也难以评估此能力。为此，我们提出InteractComp，一个旨在评估搜索智能体是否能识别查询歧义并主动交互以解决的基准。基于‘易验证、需交互澄清’原则，通过目标-干扰物方法，在9个领域构建了210个专家标注的问题，其歧义仅可通过交互消除。对17个模型的评估显示，最先进模型在无上下文时仅达13.73%准确率，而有完整上下文时为71.50%，暴露系统性过度自信而非推理缺陷。强制交互带来显著提升，表明当前策略未能激活其潜在能力。纵向分析显示，交互能力在过去15个月停滞不前，而搜索性能提升了七倍，凸显关键盲区。结合搜索任务的即时反馈特性，InteractComp成为评估与训练交互能力的重要资源。代码已开源。
原文摘要 · Abstract (English)
Language agents have demonstrated remarkable potential in web search and information retrieval. However, many search-agent benchmarks assume that user queries are complete and unambiguous. This assumption leaves under-tested a practical failure mode: agents may face ambiguous requests where the intended target cannot be identified without clarification. Yet most agents lack interactive mechanisms during the search process, and existing benchmarks cannot assess this capability. To address this gap, we introduce InteractComp, a benchmark designed to evaluate whether search agents can recognize query ambiguity and actively interact to resolve it during search. Following the principle of easy to verify, interact to disambiguate, we construct 210 expert-curated questions across 9 domains through a target-distractor methodology that creates controlled ambiguity resolvable only through interaction. Evaluation of 17 models reveals striking failure: the best model achieves only 13.73% accuracy despite 71.50% with complete context, exposing systematic overconfidence rather than reasoning deficits. Forced interaction produces dramatic gains, demonstrating latent capability current strategies fail to engage. Longitudinal analysis shows interaction capabilities stagnated over 15 months while search performance improved seven-fold, revealing a critical blind spot. This stagnation, coupled with the immediate feedback inherent to search tasks, makes InteractComp a valuable resource for both evaluating and training interaction capabilities in search agents. The code is available at https://github.com/FoundationAgents/InteractComp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。