基于残差的离线强化学习，提升数据不足场景下的策略优化效果。
Residuals-based Offline Reinforcement Learning
- 用经验残差显式建模动态预测误差，改进策略优化过程。
- 理论证明其收敛性与有限样本保证，在随机倒立摆上验证有效。
- 适合数据覆盖不全、需安全训练的高风险应用领域。
离线强化学习（Offline RL）从已有数据中学习策略，无需与真实环境交互，对高风险场景尤为重要。现有方法常依赖严格的数据覆盖假设，易受分布偏移影响。本文提出一种适用于通用状态与动作空间的残差基离线强化学习框架。通过利用经验残差，定义了一种包含动态预测误差的残差基贝尔曼最优算子，并证明其为压缩映射；在特定条件下，其不动点渐近最优并具有有限样本保证。进一步构建了残差基离线深度Q网络（DQN）算法。在随机倒立摆环境中验证了该算法的有效性。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) has received increasing attention for learning policies from previously collected data without interaction with the real environment, which is particularly important in high-stakes applications. While a growing body of work has developed offline RL algorithms, these methods often rely on restrictive assumptions about data coverage and suffer from distribution shift. In this paper, we propose a residuals-based offline RL framework for general state and action spaces. Specifically, we define a residuals-based Bellman optimality operator that explicitly incorporates estimation error in learning transition dynamics into policy optimization by leveraging empirical residuals. We show that this Bellman operator is a contraction mapping and identify conditions under which its fixed point is asymptotically optimal and possesses finite-sample guarantees. We further develop a residuals-based offline deep Q-learning (DQN) algorithm. Using a stochastic CartPole environment, we demonstrate the effectiveness of our residuals-based offline DQN algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。