实证总结大模型多轮智能体强化学习有效训练方法
A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning
- 拆解环境、奖励、策略三要素，系统分析设计选择影响
- 发现简单任务可反映模型在复杂任务的泛化能力
- 给出可复用的训练配方，适合实践者快速上手
本文研究通过多轮强化学习训练大语言模型作为智能体的实际有效方法。尽管进展迅速，现有框架和定义分散，缺乏对设计选择跨任务影响的系统分析。我们首先将设计空间分解为环境、奖励和策略三个相互关联的支柱，并基于实证得出在具身文本领域训练LLM智能体的实用配方。具体测试了TextWorld和ALFWorld（具身推理任务）以及SWE-Gym（软件工程任务）。(i) 环境方面，分析状态空间、动作空间大小及最优解长度对任务复杂度的影响，发现同一领域中简单环境即可提供泛化能力信号；(ii) 奖励方面，消融相对奖励稀疏性，发现密集回合级奖励虽加速训练，但性能与稳定性高度依赖强化学习算法；(iii) 策略方面，探讨奖励稀疏性与有偏（PPO, GRPO）和无偏（RLOO）策略梯度方法的交互关系，并在固定预算下确定最优监督微调到强化学习的训练比例。最终提炼出跨三支柱协同设计的训练配方，推动多轮智能体强化学习的研究与应用。
原文摘要 · Abstract (English)
We study what actually works and what doesn't for training large language models as agents via multi-turn reinforcement learning. Despite rapid progress, existing frameworks and definitions are fragmented, and there is no systematic formulation or analysis of which design choices matter across tasks. We address this gap by first breaking down the design space into three inter-related pillars -- environment, reward, and policy -- and empirically derive a recipe for training LLM agents in situated textual domains. In particular, we test TextWorld and ALFWorld, popular domains for testing situated embodied reasoning, as well as SWE-Gym for more software engineering style tasks. (i) For the environment, we analyze the impacts of task complexity in terms of sizes of the state and action spaces as well as optimal solution length, finding that even simple environments within a domain can provide signal on how well an agent can generalize to more complex tasks. (ii) For the reward, we ablate relative reward sparsity, observing that while dense turn-level rewards accelerate training, performance and stability is highly dependent on the choice of RL algorithm. (iii) And for the agent's policy, we explore the interplay between reward sparsity and biased (PPO, GRPO) and unbiased (RLOO) policy gradient methods in addition to showing how to find the optimal Supervised Fine-tuning (SFT) to RL training ratio given a fixed budget. We distill these findings into a training recipe that guides co-design across the three pillars, facilitating research and practical efforts in multi-turn agentic RL. Code: https://github.com/pearls-lab/meow-tea-taro
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。