arXiv:2509.21998cs.AIcs.LG2025-09

构建新基准GSM-Agent,测试大模型用工具主动搜信息解小学数学题的能力。

GSM-Agent: Understanding Agentic Reasoning Using Controllable Environments

  • 设计可控环境,让模型通过工具主动搜集解题所需信息。
  • 即使顶尖模型如GPT-5在小学数学题上也仅达67%准确率。
  • 发现多数模型不会回溯已访问信息,提出增强回溯的工具调优方法。

随着大模型作为智能体部署增多,其结合工具使用(尤其是搜索)与推理的能力——即代理推理——成为关键技能。然而,在复杂环境和任务中难以分离代理推理的表现。现有评测基准常将代理推理与高阶数学推理、专家知识等混杂。为此,我们构建了新基准GSM-Agent:要求大模型解决小学水平推理题,但仅提供问题,不给解题前提,需主动使用工具获取信息。尽管题目为小学难度,但即使是前沿模型GPT-5也仅达67%准确率。为分析推理模式,我们提出代理推理图:将环境文档嵌入聚类为节点,每次工具调用映射至最近节点,构建推理路径。令人意外的是,许多模型缺乏回溯已访问节点的能力,而该能力在静态推理中被视为关键。基于此,我们提出一种工具增强的测试时扩展方法,通过添加工具引导模型回溯,提升代理推理表现。期待该基准与框架推动未来对代理推理的理解与突破。

原文摘要 · Abstract (English)

As LLMs are increasingly deployed as agents, agentic reasoning - the ability to combine tool use, especially search, and reasoning - becomes a critical skill. However, it is hard to disentangle agentic reasoning when evaluated in complex environments and tasks. Current agent benchmarks often mix agentic reasoning with challenging math reasoning, expert-level knowledge, and other advanced capabilities. To fill this gap, we build a novel benchmark, GSM-Agent, where an LLM agent is required to solve grade-school-level reasoning problems, but is only presented with the question in the prompt without the premises that contain the necessary information to solve the task, and needs to proactively collect that information using tools. Although the original tasks are grade-school math problems, we observe that even frontier models like GPT-5 only achieve 67% accuracy. To understand and analyze the agentic reasoning patterns, we propose the concept of agentic reasoning graph: cluster the environment's document embeddings into nodes, and map each tool call to its nearest node to build a reasoning path. Surprisingly, we identify that the ability to revisit a previously visited node, widely taken as a crucial pattern in static reasoning, is often missing for agentic reasoning for many models. Based on the insight, we propose a tool-augmented test-time scaling method to improve LLM's agentic reasoning performance by adding tools to encourage models to revisit. We expect our benchmark and the agentic reasoning framework to aid future studies of understanding and pushing the boundaries of agentic reasoning.

代理推理工具使用测评基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。