提出新方法提升离线强化学习长期安全性,避免高风险行为。
Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RL
- 用哈密顿-雅可比分析生成安全标签,指导模型学习长期安全策略。
- 结合变分自编码器与悲观估计,提升对未知状态的鲁棒性与样本效率。
- 理论证明有效,实测在多个任务中显著优于现有最先进方法的安全表现。
离线安全强化学习(OSRL)从预收集数据集中学习满足约束的策略,是部署强化学习于机器人等安全关键领域的有前景方向。然而,现有方法多关注短期安全,忽视长期防护,导致在线部署时可能违反安全约束。此外,模型难以处理离线数据外的状态与动作(OOD),且样本效率低。为此,本文提出一种新框架FASP:首先利用哈密顿-雅可比(H-J)可达性分析生成可靠安全标签,作为条件变分自编码器(CVAE)与安全分类器的监督信号,实现高效采样并提供严格的长期安全保证;其次采用悲观估计方法评估奖励与代价的Q值,减少因OOD动作带来的外推误差,并通过惩罚不安全行为使智能体主动规避高风险动作;最后理论上证明该悲观估计的有效性。在DSRL基准上的大量实验表明,FASP在多个任务中表现优异,尤其在安全性方面显著超越现有最先进算法。
原文摘要 · Abstract (English)
Offline safe reinforcement learning(OSRL) derives constraint-satisfying policies from pre-collected datasets, offers a promising avenue for deploying RL in safety-critical real-world domains such as robotics. However, the majority of existing approaches emphasize only short-term safety, neglecting long-horizon considerations. Consequently, they may violate safety constraints and fail to ensure sustained protection during online deployment. Moreover, the learned policies often struggle to handle states and actions that are not present or out-of-distribution(OOD) from the offline dataset, and exhibit limited sample efficiency. To address these challenges, we propose a novel framework Feasibility-Aware offline Safe Reinforcement Learning with CVAE-based Pessimism (FASP). First, we employ Hamilton-Jacobi (H-J) reachability analysis to generate reliable safety labels, which serve as supervisory signals for training both a conditional variational autoencoder (CVAE) and a safety classifier. This approach not only ensures high sampling efficiency but also provides rigorous long-horizon safety guarantees. Furthermore, we utilize pessimistic estimation methods to estimate the Q-value of reward and cost, which mitigates the extrapolation errors induces by OOD actions, and penalize unsafe actions to enabled the agent to proactively avoid high-risk behaviors. Moreover, we theoretically prove the validity of this pessimistic estimation. Extensive experiments on DSRL benchmarks demonstrate that FASP algorithm achieves competitive performance across multiple experimental tasks, particularly outperforming state-of-the-art algorithms in terms of safety.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。