评测智能体使用工具时的错误与恢复能力，发现现有方法仍易出错。
ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems
- 构建可复现的离线测试环境，精准模拟工具调用错误
- 6800个任务中多数方法在授权受限下成功率不足30%
- 强调工具接口变化下的容错机制，适合安全验证研究者
智能体即使具备强语言理解能力，仍可能因参数错误、接口变化、恢复能力弱或重试策略低效而失败。我们提出ToolMisuseBench，一个离线确定性基准，用于评估智能体在明确步骤、调用和重试预算下的工具误用与恢复表现。该基准涵盖创建（CRUD）、检索、文件和调度环境，支持可重现的故障注入。评估指标包括成功度、无效调用行为、策略违规、恢复质量及预算效率。我们发布了包含6800个任务的公开数据集和可复现的评估流程。基线结果显示，基于模式感知的方法在特定故障上表现更优，但在发布的授权限制和硬性失败场景下，整体成功率仍较低。
原文摘要 · Abstract (English)
Tool using agents often fail for operational reasons even when language understanding is strong. Common causes include invalid arguments, interface drift, weak recovery, and inefficient retry behavior. We introduce ToolMisuseBench, an offline deterministic benchmark for evaluating tool misuse and recovery under explicit step, call, and retry budgets. The benchmark covers CRUD, retrieval, file, and scheduling environments with replayable fault injection. It reports success, invalid call behavior, policy violations, recovery quality, and budgeted efficiency. We release a public dataset with 6800 tasks and a reproducible evaluation pipeline. Baseline results show fault specific recovery gains for schema aware methods, while overall success remains limited under the released authorization and hard failure settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。