测试大模型在工具失效时的动态重规划与异常恢复能力
When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents
- 设计双维度基准，区分系统性重规划与盲目试错
- 隐式语义错误导致性能下降37%，复杂路径易陷死循环
- 模型规模提升无法缓解重规划瓶颈，适合研究鲁棒性者参考
现有基准多在理想化路径上评估大模型的工具集成推理能力，忽略真实场景中的工具失效。本文提出ToolMaze基准，用于评估大模型在工具失效下的动态路径发现与错误恢复能力。该基准采用二维设计：基于有向无环图（DAG）的拓扑复杂度，以及工具扰动的2×2分类体系（显式/隐式、瞬时/永久）。实验表明，各类扰动均显著降低模型表现，尤其在隐式语义错误下性能降幅最大，扰动恢复率（PRR）下降约37%。由于对错误输出过度信任，复杂拓扑结构导致代理陷入无效试错循环。关键发现：代理容错能力随模型规模仅提升3.66倍，远慢于基础任务执行速度，凸显动态重规划是当前未被模型扩展解决的独立瓶颈。数据与代码已公开于https://github.com/Zhudongsheng75/ToolMaze。
原文摘要 · Abstract (English)
Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures. We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents. To separate systematic replanning from blind trial-and-error, ToolMaze adopts a two-dimensional design: DAG-based topological complexity and a $2 \times 2$ taxonomy of tool perturbations (explicit/implicit, transient/permanent). Evaluations show that perturbations degrade performance across nearly all models, with the sharpest drops under implicit semantic failures. Driven by systemic over-trust in corrupted outputs, Perturbation Recovery Rate (PRR) plummets by around 37\% in these scenarios, while complex topologies trap agents in futile trial-and-error loops. Crucially, agentic fault-tolerance improves with model scale $3.66\times$ slower than basic task execution, highlighting dynamic replanning as a distinct bottleneck unaddressed by model scaling or prompting. Data and code are available at https://github.com/Zhudongsheng75/ToolMaze.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。