arXiv:2608.29834cs.CLcs.IR2026-08中稿 · EMNLP

构建对话指代定位新基准,测试智能体在真实多工具环境中的信息查找能力。

You Know What I Mean: A Benchmark for Agentic Conversational Reference Grounding

论文配图:You Know What I Mean: A Benchmark for Agentic Conversational Reference Grounding
图 1 · 摘自论文原文
  • 提出对话指代定位任务CoRG,融合对话上下文与外部工作区信息
  • 在92个仓库的400段对话中,最佳智能体仅达67.0%准确率
  • 适合研究多工具协作、智能体推理与信息验证的团队或个人

协作式对话常包含间接指代:如‘这看起来像昨天讨论的修复’,需结合对话上下文与可通过API或界面访问的外部工作区证据来理解。我们正式定义该问题为对话指代定位(CoRG):利用一组工具将对话中的指代精准定位到说话人所指的唯一外部实体。CoRG具有挑战性,因其需整合分布于对话与外部工作区的词汇、语义和时间线索。智能体必须将这些异构信号转化为有效工具使用:制定策略、发现可能候选、检查元数据与内容,并排除近似项。我们通过RepoRef基准评估CoRG,该基准包含92个仓库中400段开发者对话,涉及GitHub问题、拉取请求与提交记录。不同于单次检索任务,RepoRef常需多步工具调用。结果表明,当前智能体仍面临挑战,即使最优模型也仅达67.0%成功率,仍有三分之一指代未被解决。这些发现使CoRG成为研究智能体在真实多工具环境中搜索、检查与验证信息的可靠基准。

原文摘要 · Abstract (English)

Collaborative conversations frequently contain references whose targets are indirect rather than named: resolving "this looks like the fix discussed yesterday" requires combining conversational context with evidence from the surrounding workspace which is accessible through APIs or user interfaces. We formalize this problem as Conversational Reference Grounding (CoRG): using a given set of tools to resolve a reference in conversation to the unique external item intended by the speaker. CoRG is challenging because it combines lexical, semantic, and temporal cues distributed across the conversation and the external workspace. Agents must translate these heterogeneous signals into effective tool use: formulating strategies, discovering plausible candidates, inspecting their metadata and content, and ruling out close alternatives. We study CoRG through RepoRef, a benchmark of 400 developer-chat segments grounded in GitHub issues, pull requests, and commits across 92 repositories. Unlike single-shot retrieval tasks, RepoRef often requires multi-step tool use. Our results show that CoRG remains challenging for current agents, even the best agent reaches only 67.0% success rate, leaving one third of references unresolved. These findings position CoRG as a concrete benchmark for studying how agents search, inspect, and verify information in realistic multi-tool environments.

对话理解智能体多工具基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。