用10万+真实工作流数据训练模型，让大模型更懂复杂任务编排。
WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models
- 构建包含106,763条样本的WorkflowBench数据集，覆盖83个应用、28类任务。
- 训练出的WorkflowLlama在未见API上仍具强泛化能力，零样本测试准确率达67.4%。
- 适合需要自动化多步骤任务编排的研究者与开发者使用。
大语言模型（LLM）正推动流程自动化从机器人流程自动化向智能体流程自动化演进，但现有模型（如OpenAI GPT-4o）在工作流编排方面仍表现不足。为此，我们提出WorkflowLLM，一个以数据为中心的增强框架。首先，构建大规模微调数据集WorkflowBench，包含106,763个样本，涵盖83个应用的1,503个API，来自28个类别。数据构建分三阶段：(1) 数据收集：从Apple Shortcuts和RoutineHub提取真实工作流，转为Python风格代码，并通过ChatGPT生成层级思维链；(2) 查询扩展：利用ChatGPT生成更多任务查询，提升复杂性与多样性；(3) 工作流生成：使用标注模型生成合成任务的工作流。最终将通过质量验证的合成样本与原始样本合并，形成WorkflowBench。基于该数据集，我们微调Llama-3.1-8B得到WorkflowLlama。实验表明，WorkflowLlama具备强大复杂工作流编排能力，并在未见API上表现出显著泛化性能。此外，WorkflowBench在分布外任务规划数据集T-Eval上也展现出稳健的零样本泛化能力。数据与代码已开源：https://github.com/OpenBMB/WorkflowLLM。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have driven a revolutionary paradigm shift in process automation from Robotic Process Automation to Agentic Process Automation by automating the workflow orchestration procedure based on LLMs. However, existing LLMs (even the advanced OpenAI GPT-4o) are confined to achieving satisfactory capability in workflow orchestration. To address this limitation, we present WorkflowLLM, a data-centric framework elaborately designed to enhance the capability of LLMs in workflow orchestration. It first constructs a large-scale fine-tuning dataset WorkflowBench with 106,763 samples, covering 1,503 APIs from 83 applications across 28 categories. Specifically, the construction process can be divided into three phases: (1) Data Collection: we collect real-world workflow data from Apple Shortcuts and RoutineHub, transcribing them into Python-style code. We further equip them with generated hierarchical thought via ChatGPT. (2) Query Expansion: we prompt ChatGPT to generate more task queries to enrich the diversity and complexity of workflows. (3) Workflow Generation: we leverage an annotator model trained on collected data to generate workflows for synthesized queries. Finally, we merge the synthetic samples that pass quality confirmation with the collected samples to obtain the WorkflowBench. Based on WorkflowBench, we fine-tune Llama-3.1-8B to obtain WorkflowLlama. Our experiments show that WorkflowLlama demonstrates a strong capacity to orchestrate complex workflows, while also achieving notable generalization performance on previously unseen APIs. Additionally, WorkflowBench exhibits robust zero-shot generalization capabilities on an out-of-distribution task planning dataset, T-Eval. Our data and code are available at https://github.com/OpenBMB/WorkflowLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。