用价值函数不一致性惩罚提升离线强化学习模型精度
VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning
- 通过最小化数据直接学出的价值与模型估计价值的差异来优化模型
- 在D4RL和NeoRL上几乎所有任务均达到当前最优性能
- 适合希望提升模型准确性且可集成到现有方法中的研究者
离线强化学习从预收集数据中学习有效策略，为在线交互风险高或成本高的场景提供实用方案。基于模型的方法因其数据效率和泛化能力优势在离线强化学习中尤为突出。然而，由于模型固有误差，现有方法常依赖启发式不确定性估计引入保守性，但该方式可能不可靠。本文提出VIPO，一种新型基于模型的离线强化学习算法，通过价值估计的自监督反馈增强模型训练。具体而言，模型训练时额外最小化从离线数据直接学习的价值与模型预测价值之间的不一致性。我们在多个维度进行全面评估，结果表明VIPO能高效学习高精度模型，并在几乎所有任务上持续优于现有方法。尤其在D4RL和NeoRL基准测试中表现优异，达到当前最优水平。总体而言，VIPO提供了一个通用框架，可无缝集成至现有基于模型的离线强化学习算法中，系统性提升模型精度。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) learns effective policies from pre-collected datasets, offering a practical solution for applications where online interactions are risky or costly. Model-based approaches are particularly advantageous for offline RL, owing to their data efficiency and generalizability. However, due to inherent model errors, model-based methods often artificially introduce conservatism guided by heuristic uncertainty estimation, which can be unreliable. In this paper, we introduce VIPO, a novel model-based offline RL algorithm that incorporates self-supervised feedback from value estimation to enhance model training. Specifically, the model is learned by additionally minimizing the inconsistency between the value learned directly from the offline data and the value estimated from the model. We perform comprehensive evaluations from multiple perspectives to show that VIPO can learn a highly accurate model efficiently and consistently outperform existing methods. In particular, it achieves state-of-the-art performance on almost all tasks in both D4RL and NeoRL benchmarks. Overall, VIPO offers a general framework that can be readily integrated into existing model-based offline RL algorithms to systematically enhance model accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。