用保守策略当先验，实现强化学习安全探索。
Safe Exploration via Policy Priors
- 以保守策略为先验，结合概率动态模型乐观探索。
- 理论证明可保证学习全程安全，累积遗憾有界。
- 在仿真与真实硬件上均优于现有方法，可扩展性强。
安全探索是强化学习智能体在真实环境在线学习与适应的关键需求。本文提出SOOPER方法，利用次优但保守的策略（如来自离线数据或模拟器）作为先验。该方法通过概率动力学模型进行乐观探索，必要时悲观回退至保守策略先验。理论上证明了SOOPER在整个学习过程中保证安全，并通过有界累积遗憾实现向最优策略收敛。在多个关键安全强化学习基准和真实硬件平台上进行了大量实验，结果表明SOOPER具有良好的可扩展性，性能超越当前最先进方法，且理论保证在实践中得到验证。
原文摘要 · Abstract (English)
Safe exploration is a key requirement for reinforcement learning (RL) agents to learn and adapt online, beyond controlled (e.g. simulated) environments. In this work, we tackle this challenge by utilizing suboptimal yet conservative policies (e.g., obtained from offline data or simulators) as priors. Our approach, SOOPER, uses probabilistic dynamics models to optimistically explore, yet pessimistically fall back to the conservative policy prior if needed. We prove that SOOPER guarantees safety throughout learning, and establish convergence to an optimal policy by bounding its cumulative regret. Extensive experiments on key safe RL benchmarks and real-world hardware demonstrate that SOOPER is scalable, outperforms the state-of-the-art and validate our theoretical guarantees in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。