用新算法让小模型也能高效做多步决策，效果超大模型50%。
Reinforced Language Models for Sequential Decision Making
- 基于多步信用分配机制，优化小模型的决策路径。
- 30亿参数模型在冰湖任务上超越720亿参数基线50%。
- 适合资源有限却需复杂决策的应用场景。
大型语言模型（LLMs）具备作为序列决策代理的潜力，但其应用常受限于对大而计算成本高的模型的依赖。为改进小型模型，现有后训练方法因专为单轮交互设计，难以处理多步智能体任务中的信用分配问题。为此，我们提出多步组相对策略优化（MS-GRPO），一种基于文本中介随机博弈（TSMG）与语言-代理策略（LAP）框架的后训练算法。该方法将整个累积奖励归因于每个步骤，辅以新颖的绝对优势加权采样策略，显著提升训练性能。我们在蛇形游戏和冰湖环境上对30亿参数模型进行后训练，实验表明：该方法能有效提升决策表现——30亿参数模型在冰湖任务上的表现比720亿参数基线高出50%。结果证明，针对性后训练是构建序列决策代理的可行且高效替代方案，无需依赖模型规模。
原文摘要 · Abstract (English)
Large Language Models (LLMs) show potential as sequential decision-making agents, but their application is often limited due to a reliance on large, computationally expensive models. This creates a need to improve smaller models, yet existing post-training methods are designed for single-turn interactions and cannot handle credit assignment in multi-step agentic tasks. To address this, we introduce Multi-Step Group-Relative Policy Optimization (MS-GRPO), a new algorithm for post-training LLM agents, grounded in formal Text-Mediated Stochastic Game (TSMG) and Language-Agent Policy (LAP) frameworks. For credit assignment, MS-GRPO attributes the entire cumulative episode reward to each individual episode step. We supplement this algorithm with a novel absolute-advantage-weighted episode sampling strategy that we show improves training performance. We evaluate our approach by post-training a 3-billion parameter model on Snake and Frozen Lake. Our experiments demonstrate that the method is effective in improving decision-making performance: our post-trained 3B parameter model outperforms a 72B parameter baseline by 50% on the Frozen Lake task. This work demonstrates that targeted post-training is a practical and efficient alternative to relying on model scale for creating sequential decision-making agents using LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。