评测大模型代理在资源约束下的多工具并行调度能力。
PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents

- 构建包含资源依赖标注的可执行多工具工作流基准
- 发现良好规划不等于安全高效,资源信息能减少溢出
- 适合研究资源感知型智能体的开发者和评估者
大模型代理通过调用多个工具解决任务,平行执行对降低延迟至关重要,但安全调度难度大。现有基准主要评估工具选择、参数生成和端到端成功率,大多基于串行执行,忽视了有效的并行化与资源受限调度。这一缺失导致实际故障:串行执行安全但慢,无视资源的并行执行快但易引发可避免的资源溢出。为此,我们提出PeakBench,一个包含可执行多工具工作流、执行驱动的依赖标注和实测资源性能的基准。评估核心挑战在于归因:失败和低效可能源于错误的依赖规划、不良的资源调度,或两者兼有。PeakBench采用双维度评估框架,分离逻辑规划与物理调度,并为每项设定专用指标。实验表明,强逻辑规划无法保证资源约束下的安全高效执行;揭示资源信息可减少可避免的溢出,提升资源利用率,使PeakBench成为诊断资源感知代理行为的有效测试平台。代码已开源:https://github.com/Czzzk/Staggering-the-Peaks。
原文摘要 · Abstract (English)
LLM agents increasingly solve tasks by invoking multiple tools, where parallel execution is essential for low latency but difficult to manage safely. Existing agent benchmarks primarily evaluate tool selection, argument generation, and end-to-end success under mostly serial execution, largely overlooking valid parallelization and resource-constrained scheduling. This missing scheduling dimension creates a practical failure mode: serial execution is safe but slow, while resource-agnostic parallel execution is fast but prone to avoidable resource overflows. To address this gap, we introduce PeakBench, a benchmark of executable multi-tool workflows with execution-grounded dependency annotations and measured resource profiles. A central challenge in evaluating such workflows is attribution: failures and inefficiencies may arise from incorrect dependency planning, poor resource-constrained scheduling, or both. PeakBench addresses this challenge with a two-part evaluation framework that disentangles logical planning from physical scheduling, with dedicated metrics for each dimension. Using this framework, we show that strong logical planning does not reliably translate into safe or efficient execution under resource constraints. We further show that exposing resource information can reduce avoidable overflows and improve resource utilization, making PeakBench a useful testbed for diagnosing resource-aware agent behavior. Code is available at https://github.com/Czzzk/Staggering-the-Peaks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。