用大模型做动作引导，提升强化学习的决策能力
LaGO: Latent Action Guidance for Online Reinforcement Learning
- 让预训练大模型充当动作先验，软性指导策略优化
- 在两个基准上成功率分别提升至27.2%和15.2%
- 适合希望结合大模型知识提升在线决策的强化学习研究者
大型语言模型（LLMs）在规划与序列决策中展现出巨大潜力，但以往工作多将其直接作为控制器，依赖精确的动作生成，实践中易不可靠。本文提出面向在线强化学习的潜空间动作引导框架（LaGO），利用预训练大模型作为潜动作先验，软性引导策略优化，而非作为显式规划器或控制器。在离散控制基准CLEVR-Robot和连续控制基准Meta-World上的实验表明，LaGO持续优于基础PPO方法。具体而言，在CLEVR-Robot上平均成功率从15.1%提升至27.2%，在Meta-World上从2.7%提升至15.2%。分析进一步显示，更强的预训练大模型能提供更有效的引导，表明大模型知识可有效提升规划与在线决策性能。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown strong potential for planning and sequential decision-making, but prior work often relies on using them as direct controllers, which requires precise action generation and can be unreliable in practice. This paper proposes Latent Action Guidance for Online Reinforcement Learning (LaGO), a framework that uses a pretrained LLM as a latent action prior to softly guide online policy optimization, rather than treating the LLM as an explicit planner or controller. Experiments on both a discrete-control benchmark, CLEVR-Robot, and a continuous-control benchmark, Meta-World, demonstrate that LaGO consistently improves both reward and success rate over Vanilla PPO. In particular, LaGO increases the average success rate from 15.1% to 27.2% on CLEVR-Robot and from 2.7% to 15.2% on Meta-World. Our analysis further shows that stronger pretrained LLMs provide more effective guidance, suggesting that LLM knowledge can improve planning and online decision-making.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。