用大模型生成高质量初始数据，提升强化学习采样效率。
Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLM
- 用大模型生成覆盖最优策略的状态动作数据作为起点
- 在多个经典环境上实现最高达纯强化学习4倍的累积奖励
- 适合需要快速收敛的强化学习应用，如机器人控制
本文研究如何利用大语言模型（LLM）生成高质量数据，以暖启动强化学习（RL）算法在经典马尔可夫决策过程（MDP）环境中的学习。我们提出方法LORO，通过LLM生成覆盖最优策略所访问状态-动作的离策略数据集，随后由RL算法在环境中探索并优化该策略。实验表明，在CartPole、Pendulum等OpenAI Gym环境上，LORO不仅能收敛至最优策略，且样本效率显著优于纯LLM策略、纯强化学习及两者简单结合的基线方法，累积奖励最高可达纯强化学习基线的4倍。
原文摘要 · Abstract (English)
We investigate the usage of Large Language Model (LLM) in collecting high-quality data to warm-start Reinforcement Learning (RL) algorithms for learning in some classical Markov Decision Process (MDP) environments. In this work, we focus on using LLM to generate an off-policy dataset that sufficiently covers state-actions visited by optimal policies, then later using an RL algorithm to explore the environment and improve the policy suggested by the LLM. Our algorithm, LORO, can both converge to an optimal policy and have a high sample efficiency thanks to the LLM's good starting policy. On multiple OpenAI Gym environments, such as CartPole and Pendulum, we empirically demonstrate that LORO outperforms baseline algorithms such as pure LLM-based policies, pure RL, and a naive combination of the two, achieving up to $4 \times$ the cumulative rewards of the pure RL baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。