首次为梯度型强化学习设计安全防护机制，兼顾安全与性能。
Leveraging Analytic Gradients in Provably Safe Reinforcement Learning
- 改造可微分安全约束，适配梯度驱动的强化学习框架。
- 在3个控制任务上验证，安全训练不降低学习性能。
- 适合需高安全性保障的机器人系统开发人员。
自主机器人在安全关键场景中的应用需要安全保障。可证明安全的强化学习通过引入保护机制来提供此类保障，这些机制应在训练阶段集成以缩小仿真到现实的差距。尽管已有多种针对采样型强化学习的安全方法，但基于解析梯度的方法通常能以更少环境交互获得更优性能。然而，目前尚无适用于该范式的安全保障方案。本文填补这一空白，提出首个有效的解析梯度型强化学习安全防护方法。我们分析现有可微分保护机制，通过修改映射关系和梯度形式进行适应，并集成至先进学习算法与可微分仿真器中。在三个控制任务上的数值实验表明，采用不同保护机制后，学习过程保持安全且性能不受影响。更多可视化结果见 timwalter.github.io/safe-agb-rl.github.io。
原文摘要 · Abstract (English)
The deployment of autonomous robots in safety-critical applications requires safety guarantees. Provably safe reinforcement learning is an active field of research that aims to provide such guarantees using safeguards. These safeguards should be integrated during training to reduce the sim-to-real gap. While there are several approaches for safeguarding sampling-based reinforcement learning, analytic gradient-based reinforcement learning often achieves superior performance from fewer environment interactions. However, there is no safeguarding approach for this learning paradigm yet. Our work addresses this gap by developing the first effective safeguard for analytic gradient-based reinforcement learning. We analyse existing, differentiable safeguards, adapt them through modified mappings and gradient formulations, and integrate them into a state-of-the-art learning algorithm and a differentiable simulation. Using numerical experiments on three control tasks, we evaluate how different safeguards affect learning. The results demonstrate safeguarded training without compromising performance. Additional visuals are provided at timwalter.github.io/safe-agb-rl.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。