提出一种可证明终身安全的强化学习方法，训练与部署全程保障安全。
A Provable Approach for End-to-End Safe Reinforcement Learning
- 离线训练时用带回报条件的监督学习，部署时用高斯过程优化有限参数
- 理论证明在高概率下找到接近最优的回报设置并保证安全
- 实验显示在安全性和奖励表现上均优于现有方法，适合高风险场景
安全强化学习长期目标是确保策略在整个生命周期（从学习到运行）中始终安全。然而，现有方法难以实现这一目标。本文提出一种名为可证明终身安全强化学习（PLS）的方法，将离线安全RL与安全策略部署结合。该方法在离线阶段使用回报条件的监督学习训练策略，并在部署时通过高斯过程（GPs）谨慎优化一组称为目标回报的有限参数。理论上，我们分析了目标回报与实际回报间的数学关系，证明了使用GPs的合理性；并进一步证明，PLS能在高概率下找到近似最优的目标回报，同时保证安全。实验表明，PLS在安全性和奖励性能上均优于基线方法，实现了从学习到运行全程高收益且安全的目标。
原文摘要 · Abstract (English)
A longstanding goal in safe reinforcement learning (RL) is a method to ensure the safety of a policy throughout the entire process, from learning to operation. However, existing safe RL paradigms inherently struggle to achieve this objective. We propose a method, called Provably Lifetime Safe RL (PLS), that integrates offline safe RL with safe policy deployment to address this challenge. Our proposed method learns a policy offline using return-conditioned supervised learning and then deploys the resulting policy while cautiously optimizing a limited set of parameters, known as target returns, using Gaussian processes (GPs). Theoretically, we justify the use of GPs by analyzing the mathematical relationship between target and actual returns. We then prove that PLS finds near-optimal target returns while guaranteeing safety with high probability. Empirically, we demonstrate that PLS outperforms baselines both in safety and reward performance, thereby achieving the longstanding goal to obtain high rewards while ensuring the safety of a policy throughout the lifetime from learning to operation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。