测试智能体在工具环境不可靠时的恢复能力,发现多数表现不佳。
Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability

- 设计新基准工具,模拟五类可恢复的工具故障
- 多数智能体在可靠环境表现好,但遇故障时任务失败率飙升
- 关键短板是故障诊断和恢复策略,非计算资源不足
大型语言模型正被用作通过与外部工具环境交互来完成任务的智能体。尽管近期工具使用基准涵盖更复杂的任务场景,但仍普遍假设工具环境干净、稳定且可信,对工具环境不可靠性关注不足。我们提出 ToolBench-X,一个评估智能体在可恢复可靠性威胁下的基准。ToolBench-X 包含跨多个领域的可执行多步任务,支持顺序、并行及混合工作流,每项任务配以确定性工具和标准最终答案,实现自动评估。从无故障环境出发,注入五种结构化危害:规范漂移、调用错误、执行失败、输出漂移、跨源冲突。关键在于,每个实例均存在至少一条有效恢复路径(如重试、降级、验证或交叉检查)。实验揭示显著可靠性差距:在可靠工具下表现良好的智能体,在可恢复危害下常失效。进一步分析表明,失败更多源于有限的故障诊断能力和无效恢复策略,而非工具使用频率或推理预算。针对性恢复提示可挽救大量失败任务,而测试时扩展带来的收益有限。结果表明,工具使用评估应从函数调用准确率转向在不可靠环境中的任务完成能力。代码与数据见 https://github.com/Foreverskyou/ToolBench-X。
原文摘要 · Abstract (English)
Large language models are increasingly deployed as agents that solve tasks by interacting with external tool environments. Although recent tool-use benchmarks increasingly cover complex task settings, they still largely assume clean, stable, and trustworthy tool environments, leaving tool-environment unreliability insufficiently examined. We introduce ToolBench-X, a benchmark for evaluating agents under recoverable reliability hazards. ToolBench-X contains executable multi-step tasks across diverse domains and sequential, parallel, and mixed workflows, each paired with deterministic tools and a canonical final answer for automatic evaluation. Starting from clean tool environments, ToolBench-X injects five structured hazard types: Specification Drift, Invocation Error, Execution Failure, Output Drift, and Cross-source Conflict. Crucially, each injected instance remains solvable through at least one valid recovery path, such as retrying, fallback, verification, or cross-checking. Experiments reveal a substantial reliability gap: agents that perform well with reliable tools often fail under recoverable hazards. Further analysis shows that failures are driven less by tool-use volume or inference budget than by limited hazard diagnosis and ineffective recovery. Targeted recovery hints recover many failed tasks, while test-time scaling yields more limited gains. These results suggest that tool-use evaluation should move beyond function-call accuracy toward task completion under unreliable tool environments. The code and data is available at https://github.com/Foreverskyou/ToolBench-X.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。