构建新型智能体工作流评估基准，揭示大模型在序列与图结构规划中的能力差距。
Benchmarking Agentic Workflow Generation
- 提出WorfBench基准，支持多场景复杂图结构工作流生成
- 发现GPT-4在图规划上比序列规划低约15%性能差距
- 开源模型可泛化到未见任务，提升下游推理效率
大型语言模型（LLMs）在处理推理与规划任务中表现出色，其关键在于将复杂问题分解为可执行的工作流。现有评估框架或仅关注整体性能，或存在场景覆盖有限、工作流结构简单、评价标准宽松等问题。为此，我们提出WorfBench，一个涵盖多维度场景与复杂图结构的工作流生成基准；同时设计WorfEval，通过子序列与子图匹配算法系统性量化大模型智能体的工作流生成能力。对多种大模型的全面评估显示，智能体在序列规划与图结构规划间存在显著能力差异，即使GPT-4也存在约15%的性能差距。我们还训练了两个开源模型，并在保留任务上评估其泛化能力。此外，生成的工作流可有效提升下游任务表现，实现更快速的推理。代码与数据集已公开于https://github.com/zjunlp/WorfBench。
原文摘要 · Abstract (English)
Large Language Models (LLMs), with their exceptional ability to handle a wide range of tasks, have driven significant advancements in tackling reasoning and planning tasks, wherein decomposing complex problems into executable workflows is a crucial step in this process. Existing workflow evaluation frameworks either focus solely on holistic performance or suffer from limitations such as restricted scenario coverage, simplistic workflow structures, and lax evaluation standards. To this end, we introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures. Additionally, we present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms to accurately quantify the LLM agent's workflow generation capabilities. Through comprehensive evaluations across different types of LLMs, we discover distinct gaps between the sequence planning capabilities and graph planning capabilities of LLM agents, with even GPT-4 exhibiting a gap of around 15%. We also train two open-source models and evaluate their generalization abilities on held-out tasks. Furthermore, we observe that the generated workflows can enhance downstream tasks, enabling them to achieve superior performance with less time during inference. Code and dataset are available at https://github.com/zjunlp/WorfBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。