提出新方法提升离线强化学习在分布外区域的值函数泛化能力。
Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood
- 通过平滑邻近样本值函数来更新分布外动作的Q值
- 在D4RL上实现更准确的Q值估计并超越现有方法
- 适合关注离线RL中安全泛化与效率的研究者
离线强化学习因分布偏移导致对分布外动作的Q值过估计。现有方法虽施加约束，却常因过度保守而限制Q函数泛化，影响策略优化。本文提出在凸包及其邻域（CHN）内增强Q函数泛化的新思路。基于CHN的安全泛化保证，设计平滑贝尔曼算子（SBO），通过与邻近样本的Q值平滑融合来更新分布外Q值。理论上证明SBO可逼近CHN内所有动作的真实Q值。实际算法SQOG有效缓解过度约束问题，实现接近准确的Q值估计。在D4RL基准测试中，该方法在性能和计算效率上均优于现有先进方法。
原文摘要 · Abstract (English)
Offline Reinforcement Learning (RL) struggles with distributional shifts, leading to the $Q$-value overestimation for out-of-distribution (OOD) actions. Existing methods address this issue by imposing constraints; however, they often become overly conservative when evaluating OOD regions, which constrains the $Q$-function generalization. This over-constraint issue results in poor $Q$-value estimation and hinders policy improvement. In this paper, we introduce a novel approach to achieve better $Q$-value estimation by enhancing $Q$-function generalization in OOD regions within Convex Hull and its Neighborhood (CHN). Under the safety generalization guarantees of the CHN, we propose the Smooth Bellman Operator (SBO), which updates OOD $Q$-values by smoothing them with neighboring in-sample $Q$-values. We theoretically show that SBO approximates true $Q$-values for both in-sample and OOD actions within the CHN. Our practical algorithm, Smooth Q-function OOD Generalization (SQOG), empirically alleviates the over-constraint issue, achieving near-accurate $Q$-value estimation. On the D4RL benchmarks, SQOG outperforms existing state-of-the-art methods in both performance and computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。