用大模型和自编码器结合激励，让强化学习在奖励稀疏时更快学会任务。
LLM-Driven Intrinsic Motivation for Sparse Reward Reinforcement Learning
- 用变分自编码器奖励新状态，大模型根据环境描述生成奖励信号。
- 联合策略使智能体采样效率提升，性能远超单一方法或标准A2C。
- 适合需要高效探索的稀疏奖励场景，如复杂游戏或机器人控制。
本文探讨两种内在激励策略的结合，以提升强化学习（RL）智能体在极端稀疏奖励环境中的学习效率。传统方法因正向反馈极少而难以奏效。我们提出将变分状态内在奖励（VSIMR）与基于大语言模型（LLM）的内在奖励相结合：VSIMR利用变分自编码器（VAE）奖励状态新颖性，而LLM则借助预训练知识，根据环境和目标描述生成奖励信号，引导智能体行为。我们在MiniGrid DoorKey环境（稀疏奖励基准）中使用演员-评论家（A2C）智能体实现该联合策略。实验结果表明，相比单独使用任一策略或标准A2C（未能学习），联合策略显著提升了智能体性能与采样效率。学习曲线分析显示，VSIMR促进新状态探索，而LLM生成的奖励推动逐步向目标逼近，两者有效互补。
原文摘要 · Abstract (English)
This paper explores the combination of two intrinsic motivation strategies to improve the efficiency of reinforcement learning (RL) agents in environments with extreme sparse rewards, where traditional learning struggles due to infrequent positive feedback. We propose integrating Variational State as Intrinsic Reward (VSIMR), which uses Variational AutoEncoders (VAEs) to reward state novelty, with an intrinsic reward approach derived from Large Language Models (LLMs). The LLMs leverage their pre-trained knowledge to generate reward signals based on environment and goal descriptions, guiding the agent. We implemented this combined approach with an Actor-Critic (A2C) agent in the MiniGrid DoorKey environment, a benchmark for sparse rewards. Our empirical results show that this combined strategy significantly increases agent performance and sampling efficiency compared to using each strategy individually or a standard A2C agent, which failed to learn. Analysis of learning curves indicates that the combination effectively complements different aspects of the environment and task: VSIMR drives exploration of new states, while the LLM-derived rewards facilitate progressive exploitation towards goals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。