诊断大模型智能体规划能力，揭示其长期规划与抗干扰短板。
Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
- 构建4209个跨22领域多模态测试用例，专攻规划环节诊断。
- 发现主流模型在长程规划、工具噪声下鲁棒性差，拒答不精准。
- 可指导优化规划策略，适合评估和改进智能体系统设计者使用。
规划是大语言模型智能体的核心：行动前需分解目标、选择工具、推理约束，并判断任务是否不可行。但现有评估仅报告最终成功率，难以区分失败源于规划还是执行。本文提出代理规划基准（APB），包含4209个多模态案例，覆盖22个领域与五种设置，涵盖整体规划、反馈驱动的分步规划，以及在冗余工具、故障工具和无解任务下的鲁棒性。在12个主流多模态大模型上测试显示，普遍存在长程规划缺陷、工具噪声鲁棒性弱、拒绝响应不校准及推理时修正能力不足。进一步在200个ToolSandbox任务和200个τ²-bench任务中验证，基于APB的规划优化显著提升计划正确率、计划评分与下游执行指标。因此，APB可作为执行评估的上游诊断补充。代码与数据集已开源。
原文摘要 · Abstract (English)
Planning is central to LLM agents: before acting, an agent must decompose goals, select tools, reason over constraints, and decide when a task is infeasible. Yet existing agent evaluations often report only end-to-end success, making it difficult to determine whether failures stem from planning or execution. We introduce Agent Planning Benchmark (APB), a planning-specific diagnostic benchmark with 4,209 multimodal cases across 22 domains and five settings, covering holistic planning, feedback-conditioned step-wise planning, and robustness under extraneous tools, broken tools, and unsolvable tasks. Across 12 MLLMs, APB reveals systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement. We further validate APB on 200 ToolSandbox tasks and 200 $τ^2$-bench tasks, where APB-guided refinement consistently improves plan correctness, plan grade, and downstream execution metrics across three representative models. APB thus serves as an upstream diagnostic complement to execution benchmarks. The APB benchmark and code are available in \href{https://github.com/Mikivishy/AgentPlanningBenchmark}{this URL}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。