用单轮强化学习训练大模型，实现复杂多步任务规划
Training Task Reasoning LLM Agents for Multi-turn Task Planning via Single-turn Reinforcement Learning
- 将多轮任务规划转为单轮推理问题，用密集奖励优化策略
- 1.5B模型在长周期任务上达70%成功率，优于14B基线模型
- 适合需要高效任务规划的自动化系统开发者
大型语言模型在知识获取、推理和工具使用方面展现出强大能力，是构建自主智能体的有力候选。然而，复杂多轮任务规划的训练面临奖励稀疏、信用分配困难以及多轮交互中强化学习计算开销大的挑战。为此，本文提出一种新方法，将多轮任务规划转化为单轮任务推理问题，通过专家轨迹提供的密集且可验证奖励，利用分组相对策略优化（GRPO）实现高效策略优化。理论分析表明，单轮任务推理中GRPO的提升能保证最小轮次下多轮任务成功率的下界，并可推广至更短时序的子任务。在复杂任务规划基准上的实验表明，仅用单轮GRPO训练的1.5B参数模型，在长周期任务上取得70%的成功率，性能优于最大达14B参数的基线模型。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capabilities in knowledge acquisition, reasoning, and tool use, making them promising candidates for autonomous agent applications. However, training LLM agents for complex multi-turn task planning faces significant challenges, including sparse episode-wise rewards, credit assignment across long horizons, and the computational overhead of reinforcement learning in multi-turn interaction settings. To this end, this paper introduces a novel approach that transforms multi-turn task planning into single-turn task reasoning problems, enabling efficient policy optimization through Group Relative Policy Optimization (GRPO) with dense and verifiable reward from expert trajectories. Our theoretical analysis shows that GRPO improvement on single-turn task reasoning results in a lower bound of the multi-turn success probability under the minimal turns, as well as the generalization to subtasks with shorter horizons. Experimental evaluation on the complex task planning benchmark demonstrates that our 1.5B parameter model trained with single-turn GRPO achieves superior performance compared to larger baseline models up to 14B parameters, with success rates of 70% for long-horizon planning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。