arXiv:2602.01202cs.AI2026-02被引 5

让大模型分步思考并动态调整任务流程,提升复杂问题解决能力。

Workflow-R1: Group Sub-sequence Policy Optimization for Multi-turn Workflow Construction

  • 将任务构建改为多轮自然语言对话式决策,更贴近人类思考过程。
  • 在多个问答数据集上超越现有方法,准确率显著提升。
  • 适合需要分步推理的自动化系统设计者或研究者使用。

基于大模型的智能体在处理复杂推理任务中展现出强大性能。然而,现有工作通常将流程生成视为静态的一次性代码生成问题,对模型编码能力要求过高,限制了动态求解灵活性。本文提出 Workflow-R1 框架,将流程构建重构为多轮、基于自然语言的序列决策过程。为解决多轮交互中的优化粒度不匹配问题,引入分组子序列策略优化(GSsPO)。该方法明确适配智能体推理中“思考-行动”交替的动态机制,本质上是一种结构感知的强化学习算法,可泛化至多种多轮智能体序列决策任务。通过将优化单元校准为原子“思考-行动”循环,使梯度更新与语义边界对齐,确保复杂多轮推理任务中的稳健学习。在多个 QA 基准上的大量实验表明,Workflow-R1 显著优于基线方法,验证了 GSsPO 作为通用序列推理解决方案的有效性,并确立 Workflow-R1 作为自动化流程优化的新范式。

原文摘要 · Abstract (English)

The rapid evolution of agentic workflows has demonstrated strong performance of LLM-based agents in addressing complex reasoning tasks. However, existing workflow optimization methods typically formulate workflow synthesis as a static, one-shot code-centric generation problem. This paradigm imposes excessive constraints on the model's coding capabilities and restricts the flexibility required for dynamic problem-solving. In this paper, we present Workflow-R1, a framework that reformulates workflow construction as a multi-turn, natural language-based sequential decision-making process. To resolve the optimization granularity mismatch inherent in such multi-turn interactions, we introduce Group Sub-sequence Policy Optimization (GSsPO). While explicitly tailored to align with the interleaved Think-Action dynamics of agentic reasoning, GSsPO fundamentally functions as a structure-aware RL algorithm generalizable to a broad class of multi-turn agentic sequential decision-making tasks. By recalibrating the optimization unit to the composite sub-sequence, specifically the atomic Think-Action cycle, it aligns gradient updates with the semantic boundaries of these interactions, ensuring robust learning in complex multi-turn reasoning tasks. Through extensive experiments on multiple QA benchmarks, Workflow-R1 outperforms competitive baselines, validating GSsPO as a generalized solution for sequential reasoning and establishing Workflow-R1 as a promising new paradigm for automated workflow optimization.

智能体多轮推理强化学习流程优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。