评测大模型在灾情应对工具链中的精准规划能力，揭示语义理解与执行约束间的差距。
DisasterBench: Benchmarking LLM Planning under Typed Tool Interface Constraints
- 构建灾情响应工具链的多智能体规划基准，强调参数绑定与依赖传递的正确性
- 发现工具错配和参数错误是失败首要原因，占首因故障超60%
- 指出冗长推理易与结构化输出冲突，适合需精确执行的系统设计者参考
灾害带来严重社会影响，亟需快速协调多种异构人工智能工具（如卫星分析、洪水预测、损毁评估）形成连贯的多步骤工作流。随着大语言模型越来越多地担任此类流水线的调度者，有效协同不仅要求选择语义合理的工具，更需生成具备正确参数绑定和依赖传播的可执行工作流。我们提出DisasterBench，一个用于评估在语义相似但操作不同的灾情应对工具接口约束下，结构化多智能体规划能力的基准测试。为实现步骤级失败归因，我们进一步提出首个失败点（FPoF）机制，定位预测工作流中最早的根因，分离主错误与后续级联效应。评估结果显示：规划方法的有效性高度依赖模型能力；工具错配和参数绑定错误主导首次失败，揭示语义对齐与执行一致性是两个独立瓶颈；冗长的中间推理会与结构化输出要求产生指令冲突，破坏计划生成。这些发现凸显了语义推理与执行导向协调之间的根本差距，强调需要同时建模语义意图、执行约束与工作流一致性的规划框架。代码、数据与评估资源已公开于：https://github.com/TamuChen18/DisasterBench_Open
原文摘要 · Abstract (English)
Disasters cause severe societal impacts, demanding rapid coordination of heterogeneous AI tools, from satellite analysis to flood prediction and damage assessment, into coherent multi-step workflows. As LLMs increasingly serve as orchestrators of such pipelines, effective coordination requires more than selecting semantically plausible tools: LLMs must generate executable workflows with correct parameter binding and dependency propagation. We introduce DisasterBench, a benchmark for evaluating structured multi-agent planning over semantically similar but operationally distinct disaster-response tools. To enable step-level failure attribution, we further propose First-Point-of-Failure (FPoF), which localizes the earliest root cause in a predicted workflow, separating primary errors from downstream cascading effects. Our evaluation reveals three findings: planning method effectiveness depends strongly on model capacity; tool mismatch and parameter-binding errors dominate first failures, revealing semantic grounding and execution consistency as distinct bottlenecks; and verbose intermediate reasoning can create instruction clash with structured output requirements, disrupting plan generation. Together, these findings highlight a fundamental gap between semantic reasoning and execution-grounded coordination, underscoring the need for planning frameworks that jointly model semantic intent, execution constraints, and workflow consistency. Code, data, and evaluation resources are available at: https://github.com/TamuChen18/DisasterBench_Open
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。