arXiv:2608.23635cs.SEcs.AI2026-08

为工具调用智能体设计分阶段故障诊断基准,精准定位失败环节。

ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents

论文配图:ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents
图 1 · 摘自论文原文
  • 按工具调用流程分阶段注入四类扰动,实现故障溯源。
  • 15,456次实验显示输出扰动是主要瓶颈,鲁棒性下降显著。
  • 适合研究模型鲁棒性、工具调用可靠性与系统级诊断的开发者。

大语言模型依赖工具调用作为核心能力,可调用外部系统完成文本生成之外的任务。然而,纯端到端成功无法识别故障来源或传播路径。我们提出ToolRobustBench,一个面向工具调用智能体的分阶段诊断基准。该基准将四类扰动(工具接口、用户意图、工具输出/观测、运行环境)与工具调用流程对齐,定位失败于工具选择、模式对齐、参数绑定、输出/反馈处理及端到端任务成功等环节。在7个模型、16个本地工具、4类扰动、14种子类型上共执行15,456次单家族实例实验,结果表明:清洁性能高但分布不均,鲁棒性显著下降,其中工具输出/观测扰动为最主要瓶颈。混合扰动实验揭示非加性失败模式,无法由单一扰动结果解释。ToolRobustBench提供了确定性且链式感知的基准,超越清洁准确率,实现系统性故障诊断。

原文摘要 · Abstract (English)

Large language models (LLMs) rely on tool calling as a fundamental agent capability, enabling them to invoke external systems and complete tasks beyond text generation. However, clean end-to-end (E2E) success cannot identify where a tool-use failure originates or how it propagates through a call. We introduce ToolRobustBench, a stage-wise diagnostic benchmark for tool-calling agents, where a tool-calling agent is an LLM system that selects a tool, supplies structured arguments, and interprets its returned feedback. ToolRobustBench aligns four perturbation families with the tool-use pipeline: tool-interface, user-intent, tool-output/observation, and runtime-environment perturbations. It attributes failures to tool selection, schema grounding, argument binding, tool-output/runtime-feedback handling, and E2E task success. Experiments on 15,456 single-family instances across 7 models, 16 sampled local tools, 4 perturbation families, and 14 subtypes show high but non-uniform clean performance and substantial robustness degradation, with tool-output/observation perturbation the dominant bottleneck. Mixed-family experiments reveal non-additive failure patterns that are not explained by isolated single-family results. Thus, ToolRobustBench provides a deterministic and cascade-aware benchmark for diagnosing robustness beyond clean tool-calling accuracy;

工具调用鲁棒性评估故障诊断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。