arXiv:2608.01358cs.CL2026-08

首个针对多跳搜索中拒绝行为的诊断基准,揭示大模型在复杂推理中的失败模式。

HopRefusalBench: Diagnosing Refusal Failures in Search-Augmented Agents for Multi-Hop Reasoning

论文配图:HopRefusalBench: Diagnosing Refusal Failures in Search-Augmented Agents for Multi-Hop Reasoning
图 1 · 摘自论文原文
  • 构建889个基于KILT实体路径的不可回答问题,覆盖三种未答原因与三类问题结构。
  • 顶尖模型仅42.9%正确停止,多数拒绝响应虽理由正确但未适切拒绝。
  • 发现根节点和中间节点问题最难,模型常陷入幻觉或耗尽搜索预算。

搜索增强型大语言模型在解决知识密集型任务方面日益强大,但当多跳问题本质上无法回答时,其行为仍不清晰。现有拒绝基准主要暴露单跳查询表面缺陷,无法揭示经有效中间推理与检索后才显现的失败。我们提出HopRefusalBench,首个可控的多跳搜索拒绝诊断基准,包含889个基于KILT实体路径构造的不可回答问题,涵盖答案未知、前提错误、上下文不明确三种原因,与根节点、中间节点、终端节点三类拓扑结构,使前提验证、中间桥接验证、终端终止行为可分别观测。我们进一步提出涵盖目标感知拒绝、伪拒绝、幻觉完成、搜索预算耗尽的最终结果分类体系,并设计源感知轨迹度量用于分析触发后的延续行为与令牌浪费。在十种前沿专有及开源模型的搜索增强模式下,最优模型的目标感知正确终止率(TCHR)仅为42.9%。根节点与中间节点问题始终更难,所有模型在错误前提问题上表现最佳,在不明确问题上最差。然而,跨类别汇总后,各模型84.7%–98.4%的拒绝类响应能识别正确理由,表明主要瓶颈在于未能作出适当非回答;失败轨迹则转向幻觉或搜索预算耗尽。这些结果确立了多跳搜索中的拒绝为关键评估问题,并为诊断与提升搜索增强代理的可靠性提供基础。

原文摘要 · Abstract (English)

Search-augmented large language model agents are increasingly capable of solving knowledge-intensive tasks, but their behavior when a multi-hop question is fundamentally unanswerable remains poorly understood. Existing abstention benchmarks largely expose defects at the surface of single-hop queries and therefore cannot reveal failures that emerge only after valid intermediate reasoning and retrieval. We introduce HopRefusalBench, the first controlled benchmark of refusal within multi-hop search, comprising 889 unanswerable questions constructed from KILT-grounded entity paths. It crosses three causes of unanswerability (answer unknown, false premise, and underspecified context) with root, middle, and terminal topologies, making premise verification, intermediate-bridge validation, and terminal stopping separately observable. We further propose a final-outcome taxonomy spanning target-aware refusal, pseudo-refusal, hallucinated completion, and search-budget exhaustion, together with source-aware trajectory metrics for post-trigger continuation and token waste. Across ten frontier proprietary and open-weight models in search-augmented mode, the best model achieves a target-aware correct halting rate (TCHR) of only 42.9%. Root and middle items are consistently harder than terminal items, and all models attain their highest TCHR on false premises and their lowest on underspecified questions. Yet when pooled across categories, 84.7--98.4% of each model's explicit refusal-like responses identify the correct rationale, localizing the main bottleneck to committing to an appropriate non-answer; failed trajectories instead diverge into hallucination or search-budget exhaustion. These results establish refusal in multi-hop search as a consequential evaluation problem and provide a foundation for diagnosing and improving the reliability of search-augmented agents.

多跳推理拒绝机制评估基准搜索增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。