arXiv:2602.21265cs.CLcs.LG2026-02

构建可控工具库的数学推理测试集,评估模型长序列工具使用能力。

ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints

  • 将数学解题步骤转为带自然语言描述的可复用工具
  • 通过相似度分级干扰项控制工具库难度,量化模型适应性与鲁棒性
  • 诊断模型在工具链中断、干扰项影响下的失败模式

我们提出 oolmath,一个基于数学问题的诊断性基准,用于在可控工具库条件下评估长时序工具使用能力。 oolmath 将 MATH 数据集中的分步解题过程转化为带有自然语言描述和类型定义的可复用 Python 工具,并为每个问题构建需顺序调用、中间结果复用及逻辑连贯的工具调用链环境。通过构建黄金工具与相似度分级的干扰项, oolmath 可控调节工具可用性和库复杂度。引入行为感知评估指标,实现超越最终准确率的诊断分析。重点考察三个维度:(1) 适应性——当黄金工具被干扰项完全替换后,原成功率保留程度;(2) 鲁棒性——添加干扰项作为噪声时的表现稳定性;(3) 工具连通性——长工具链执行中保持准确性的能力。结合追踪级失败分析,揭示模型表现出可靠工具使用、工具回避、自适应替代以及不可靠工具库影响等不同特征。总体而言, oolmath 提供了一个受控测试平台,用于评估语言模型在工具可用性变化、干扰项存在及长轨迹任务中维持正确性的能力。

原文摘要 · Abstract (English)

We introduce \ToolMATH, a math-grounded diagnostic benchmark for evaluating long-horizon tool use under controllable tool-catalog conditions. \ToolMATH converts stepwise MATH solutions into reusable Python tools with natural-language descriptions and typed schemas, and pairs each problem with a tool environment requiring sequential tool use, intermediate-output reuse, and logically connected tool-call chains. \ToolMATH controls tool availability and catalog difficulty by constructing gold tools and graded distractors with varying similarity to gold tools. \ToolMATH also incorporates behavior-conditioned metrics, enabling diagnostic evaluation beyond final accuracy. Building on these measurements, \ToolMATH emphasizes three evaluation axes: (1) \emph{Adaptability} measures how much Gold-only success is retained when gold tools are replaced entirely by distractors; (2) \emph{Robustness} measures stability under adding distractors as a noise; and (3) \emph{Tool Connectivity} measures whether models preserve accuracy over long executed tool-call chains. Furthermore, trace-level failure analyses characterize how models fail under each tool-catalog condition. Together, these diagnostics reveal distinct model profiles: reliable tool use, tool avoidance, adaptive substitution, and impacts of unreliable tool catalogs. Overall, \ToolMATH provides a controlled testbed for evaluating how language models adapt to changing tool availability, remain robust to distractors, and maintain correctness across long-horizon tool-use trajectories.

工具使用长序列推理诊断评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。