通过决策点限制改进范围，实现安全高效的批量强化学习。
Decision-Point Guided Safe Policy Improvement
- 仅在高频访问的状态-动作区域（决策点）进行策略优化
- 数据依赖的置信边界不随状态动作空间增大而恶化
- 适合对安全性要求高且数据稀疏的真实场景
在批量强化学习中，安全策略改进（SPI）旨在确保学习到的策略性能不低于生成数据集的行为策略。核心挑战在于，在许多状态-动作对访问频率较低的情况下平衡改进与风险。本文提出决策点强化学习（DPRL），通过限制可改进的状态-动作对集合（或连续状态下的区域），确保在密集访问状态（即决策点）中实现高置信度改进，同时仍利用稀疏访问状态的数据。通过合理限制偏离行为策略的范围和方式，本方法获得比之前工作更紧的边界；具体而言，其数据依赖的边界不随状态空间和动作空间规模增长。除理论分析外，我们在合成与真实数据集上验证了DPRL的安全性与性能。
原文摘要 · Abstract (English)
Within batch reinforcement learning, safe policy improvement (SPI) seeks to ensure that the learnt policy performs at least as well as the behavior policy that generated the dataset. The core challenge in SPI is seeking improvements while balancing risk when many state-action pairs may be infrequently visited. In this work, we introduce Decision Points RL (DPRL), an algorithm that restricts the set of state-action pairs (or regions for continuous states) considered for improvement. DPRL ensures high-confidence improvement in densely visited states (i.e. decision points) while still utilizing data from sparsely visited states. By appropriately limiting where and how we may deviate from the behavior policy, we achieve tighter bounds than prior work; specifically, our data-dependent bounds do not scale with the size of the state and action spaces. In addition to the analysis, we demonstrate that DPRL is both safe and performant on synthetic and real datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。