诊断大模型用工具时的失败模式，揭示高分模型背后的真实问题
ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
- 设计1000个任务，区分需用工具与无需用工具的情况，精准定位错误类型
- 19个主流模型中最佳仅86.33%正确使用工具，且同水平模型失败方式差异大
- 发现模型存在盲目调用工具或忽略结果等典型问题，适合评估和改进智能体
工具调用是现代语言模型智能体的核心能力，但综合基准得分常掩盖工具使用失败的具体位置。一个从不调用所需工具的模型与一个调用了工具却忽略结果的模型，在最终任务准确率上可能表现相似。我们提出ToolFailBench，一个诊断性基准，涵盖金融、医疗、法律、网络安全和房地产领域的1000个任务。工具必需任务返回模型无法猜测的值，迫使模型依赖工具；控制任务则附加相同工具但应直接作答。我们通过规则分类器与两名LLM裁判（多数投票）标注每条执行轨迹的失败类型：工具跳过、结果忽略、输出捏造、非必要工具调用。在19个主流模型中，最佳模型的纯净工具使用率为86.33%，表明忠实工具使用尚未饱和。更重要的是，具有相似综合得分的模型以不同方式失败：多数模型在无工具控制任务中表现稳健，而Llama-3.1系列表现出‘总是调用’倾向；在相同参数规模下，Llama-3.1-70B与Qwen2.5-72B在控制任务准确率上相差89个百分点。工具使用评估不仅应关注是否调用工具，更应考察是否正确使用输出，并避免在无需时调用。
原文摘要 · Abstract (English)
Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。