利用参数化结构提升离线安全策略优化的数据效率
Data-Efficient Safe Policy Improvement Using Parametric Structure
- 基于参数依赖关系改进动态估计，用相同数据更准预测转移
- 通过博弈抽象剪枝冗余动作，显著降低学习复杂度
- 结合SMT求解进一步识别可剪枝动作，适合高可靠性场景
安全策略改进（SPI）是离线强化学习问题，目标是在仅使用数据集和行为策略的前提下，以高置信度计算出性能优于行为策略的新策略。马尔可夫决策过程（MDPs）是此类问题的标准建模工具。在许多实际应用中，环境转移动态中分布间的参数化依赖关系是已知的。本文通过三项贡献提升SPI的数据效率：（1）提出一种参数化SPI算法，利用分布间的已知相关性，在相同数据量下更准确地估计转移动态；（2）设计一种基于博弈的抽象预处理技术，剪枝环境中冗余动作；（3）提出一种基于可满足性模理论（SMT）求解的进阶预处理方法，能识别更多可剪枝动作。实验与消融研究显示，这些方法使SPI的数据效率提升多个数量级，同时保持相同的可靠性保证。
原文摘要 · Abstract (English)
Safe policy improvement (SPI) is an offline reinforcement learning problem in which a new policy that reliably outperforms the behavior policy with high confidence needs to be computed using only a dataset and the behavior policy. Markov decision processes (MDPs) are the standard formalism for modeling environments in SPI. In many applications, additional information in the form of parametric dependencies between distributions in the transition dynamics is available. We make SPI more data-efficient by leveraging these dependencies through three contributions: (1) a parametric SPI algorithm that exploits known correlations between distributions to more accurately estimate the transition dynamics using the same amount of data; (2) a preprocessing technique that prunes redundant actions from the environment through a game-based abstraction; and (3) a more advanced preprocessing technique, based on satisfiability modulo theory (SMT) solving, that can identify more actions to prune. Empirical results and an ablation study show that our techniques increase the data efficiency of SPI by multiple orders of magnitude while maintaining the same reliability guarantees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。