改进离线强化学习的值函数外推问题，提升复杂任务表现
Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
- 通过奖励缩放与不可行动作惩罚，引导值函数在数据外渐变衰减
- 在D4RL基准上优于主流算法，尤其在AntMaze Ultra任务中显著领先
- 适合需要稳定离线训练的强化学习研究者与工业应用开发者
基于离线数据的强化学习面临值函数外推误差问题。我们首先证明：在数据范围外进行线性外推会引发严重偏差。为缓解此问题，提出一种在数据范围外逐步降低Q值的机制，通过层归一化下的奖励缩放（RS-LN）和不可行动作惩罚（PA）实现。二者结合形成新算法PARS。在D4RL基准上评估，PARS在离线训练和在线微调中均优于现有先进方法，尤其在具有挑战性的AntMaze Ultra任务中表现突出。
原文摘要 · Abstract (English)
Reinforcement learning with offline data suffers from Q-value extrapolation errors. To address this issue, we first demonstrate that linear extrapolation of the Q-function beyond the data range is particularly problematic. To mitigate this, we propose guiding the gradual decrease of Q-values outside the data range, which is achieved through reward scaling with layer normalization (RS-LN) and a penalization mechanism for infeasible actions (PA). By combining RS-LN and PA, we develop a new algorithm called PARS. We evaluate PARS across a range of tasks, demonstrating superior performance compared to state-of-the-art algorithms in both offline training and online fine-tuning on the D4RL benchmark, with notable success in the challenging AntMaze Ultra task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。