新算法让强化学习在不确定环境下更安全，大幅减少失败次数。
Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty
- 用可扩展的安全过滤器和不确定性感知模型联合训练策略
- 在双足行走和倒立摆任务中，失败率降低两个数量级
- 适合高维系统且无需太多先验知识，尤其适合实际部署
安全仍是强化学习中的关键挑战，尤其在训练阶段。虽然安全过滤器有助于安全探索，但通常难以适用于高维未知动态系统。我们提出一种新型算法Dyna-SAuR，通过学习不确定性感知的动力学模型，同时训练可扩展的安全过滤器与控制策略，且仅需极少领域知识。该过滤器能避开失败和高不确定性区域，随着模型改进，安全状态集扩大，从而降低滤波器的保守性。在目标达成型倒立摆和MuJoCo Walker任务中，Dyna-SAuR相较于现有方法将失败次数减少了两个数量级，验证了其有效性。
原文摘要 · Abstract (English)
Safety remains an open problem in reinforcement learning (RL), especially during training. While safety filters are promising to address safe exploration, they are generally poorly suited for high-dimensional systems with unknown dynamics. We propose Dyna-style Safety Augmented Reinforcement Learning (Dyna-SAuR), a novel algorithm that learns both a scalable safety filter and a control policy using a learned uncertainty-aware dynamics model, while requiring minimal domain knowledge. The filter avoids failures and high uncertainty regions. Thus, better models expand the set of safe and certain states, reducing filter conservatism. We present the effectiveness of Dyna-SAuR on goal-reaching CartPole as well as MuJoCo Walker, reducing failures compared to state-of-the-art methods by 2 orders of magnitude.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。