提出新评估框架，平衡离线强化学习中的保守性与性能
Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
- 用时序差分误差+行为克隆结合改进贝尔曼更新
- 在基准数据集上优于主流离线RL算法
- 适合需要稳定又高效学习的离线强化学习场景
离线强化学习旨在仅从静态数据集学习最优策略而无需与环境交互。其主要挑战在于学习策略与行为策略间的分布偏移，导致分布外（OOD）动作和价值过估计。为防止严重过估计，价值函数需保持保守；但过度保守会抑制性能提升。为此，我们提出温和保守正则化评估（MCRE）框架，通过在贝尔曼备份中结合时序差分（TD）误差与行为克隆项，实现保守性与性能的平衡。基于此，我们开发了温和保守正则化Q学习（MCRQ）算法，将MCRE融入非策略演员-评论家框架。实验表明，MCRQ在基准数据集上超越强基线及当前最先进离线强化学习算法。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) seeks to learn optimal policies from static datasets without further environment interaction. A key challenge is the distribution shift between the learned and behavior policies, leading to out-of-distribution (OOD) actions and overestimation. To prevent gross overestimation, the value function must remain conservative; however, excessive conservatism may hinder performance improvement. To address this, we propose the mildly conservative regularized evaluation (MCRE) framework, which balances conservatism and performance by combining temporal difference (TD) error with a behavior cloning term in the Bellman backup. Building on this, we develop the mildly conservative regularized Q-learning (MCRQ) algorithm, which integrates MCRE into an off-policy actor-critic framework. Experiments show that MCRQ outperforms strong baselines and state-of-the-art offline RL algorithms on benchmark datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。