提出SWEET-RL算法，让大模型在协作编程中更高效地完成多轮任务。
SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks
- 通过训练时信息设计分步奖励机制，实现多轮交互中的精准信用分配。
- 在新基准ColBench上，成功率和胜率提升6个百分点，优于现有方法。
- 适合研究多轮强化学习、大模型协作任务的开发者与研究人员。
大型语言模型（LLM）代理需要在真实任务中进行多轮交互。然而，现有的多轮强化学习（RL）算法在优化LLM代理时难以有效进行多轮信用分配，且未充分利用LLM的泛化能力，如何构建此类算法仍不明确。为此，我们首先引入一个新基准ColBench，其中LLM代理与人类协作，通过多轮交互解决后端编程和前端设计等实际任务。基于该基准，我们提出一种新型强化学习算法SWEET-RL（基于训练时信息的分步评估），利用精心设计的优化目标，使评论模型能访问额外的训练时信息，从而为策略模型提供分步奖励。实验表明，SWEET-RL在ColBench上相比其他最先进的多轮RL算法，成功率达6%绝对提升，胜率亦显著提高，使Llama-3.1-8B在真实协作内容生成任务中达到或超越GPT4-o的性能。
原文摘要 · Abstract (English)
Large language model (LLM) agents need to perform multi-turn interactions in real-world tasks. However, existing multi-turn RL algorithms for optimizing LLM agents fail to perform effective credit assignment over multiple turns while leveraging the generalization capabilities of LLMs and it remains unclear how to develop such algorithms. To study this, we first introduce a new benchmark, ColBench, where an LLM agent interacts with a human collaborator over multiple turns to solve realistic tasks in backend programming and frontend design. Building on this benchmark, we propose a novel RL algorithm, SWEET-RL (RL with Step-WisE Evaluation from Training-time information), that uses a carefully designed optimization objective to train a critic model with access to additional training-time information. The critic provides step-level rewards for improving the policy model. Our experiments demonstrate that SWEET-RL achieves a 6% absolute improvement in success and win rates on ColBench compared to other state-of-the-art multi-turn RL algorithms, enabling Llama-3.1-8B to match or exceed the performance of GPT4-o in realistic collaborative content creation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。