arXiv:2506.08295cs.LGcs.AI2025-06ICML被引 23

测试大模型主动提问获取信息的能力,发现其表现远低于被动推理。

From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?

论文配图:From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?
图 1 · 摘自论文原文
  • 设计新基准AR-Bench,让模型通过提问主动获取缺失信息
  • 现有大模型在主动推理任务中失败率高,难以获取关键信息
  • 适合研究智能体交互、主动学习的学者使用

现有评测主要考察大语言模型(LLMs)在信息完备条件下的被动推理能力,而对需要与外部系统交互以获取缺失证据的主动推理关注不足。为此,我们提出AR-Bench,一个专为评估主动推理能力设计的新基准。该基准包含侦探案件、情境谜题和猜数三类任务,模拟真实世界代理场景,涵盖常识、逻辑与符号推理挑战。实证评估显示,当前主流LLMs在主动推理中表现不佳,常无法有效获取或利用所需信息,暴露出其被动与主动推理能力间的显著差距。消融实验表明,即使采用基于树的搜索或后训练等先进策略,提升也有限,难以满足实际部署需求。研究呼吁发展更有效的主动推理方法,如交互式学习、实时反馈机制与环境感知目标。基准代码已开源:https://github.com/tmlr-group/AR-Bench。

原文摘要 · Abstract (English)

While existing benchmarks probe the reasoning abilities of large language models (LLMs) across diverse domains, they predominantly assess passive reasoning, providing models with all the information needed to reach a solution. By contrast, active reasoning-where an LLM must interact with external systems to acquire missing evidence or data-has received little systematic attention. To address this shortfall, we present AR-Bench, a novel benchmark designed explicitly to evaluate an LLM's active reasoning skills. AR-Bench comprises three task families-detective cases, situation puzzles, and guessing numbers-that together simulate real-world, agentic scenarios and measure performance across commonsense, logical, and symbolic reasoning challenges. Empirical evaluation on AR-Bench demonstrates that contemporary LLMs exhibit pronounced difficulties with active reasoning: they frequently fail to acquire or leverage the information needed to solve tasks. This gap highlights a stark divergence between their passive and active reasoning abilities. Moreover, ablation studies indicate that even advanced strategies, such as tree-based searching or post-training approaches, yield only modest gains and fall short of the levels required for real-world deployment. Collectively, these findings highlight the critical need to advance methodology for active reasoning, e.g., incorporating interactive learning, real-time feedback loops, and environment-aware objectives for training. The benchmark is publicly available at: https://github.com/tmlr-group/AR-Bench.

主动推理大模型评测智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。