用成本调节奖励，让强化学习更安全。
Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards
- 引入成本估计算法，实时评估潜在风险。
- 在多个环境上实现更高安全性能，优于主流方法。
- 适合需高安全性的机器人、自动驾驶场景。
安全强化学习旨在训练智能体在真实环境中提升性能的同时遵守安全约束，因为超出安全限制可能带来严重后果。本文提出一种新型安全强化学习方法——安全调制策略优化（SMPO），通过安全调制奖励，在标准策略优化框架内实现安全策略学习。具体地，将安全违规成本视为与奖励并行的环境反馈，并引入Q-cost函数作为安全评判器，估计未来累积成本。随后设计成本感知加权函数，根据安全评判器的估计结果调节奖励，确保满足安全限制的同时最大化期望回报。策略函数与安全评判器通过在线交互中的梯度下降同步学习。在多个强化学习环境中进行实验，结果表明该方法在整体安全强化学习性能上优于多种经典及前沿对比方法。
原文摘要 · Abstract (English)
Safe Reinforcement Learning (Safe RL) aims to train an RL agent to maximize its performance in real-world environments while adhering to safety constraints, as exceeding safety violation limits can result in severe consequences. In this paper, we propose a novel safe RL approach called Safety Modulated Policy Optimization (SMPO), which enables safe policy function learning within the standard policy optimization framework through safety modulated rewards. In particular, we consider safety violation costs as feedback from the RL environments that are parallel to the standard awards, and introduce a Q-cost function as safety critic to estimate expected future cumulative costs. Then we propose to modulate the rewards using a cost-aware weighting function, which is carefully designed to ensure the safety limits based on the estimation of the safety critic, while maximizing the expected rewards. The policy function and the safety critic are simultaneously learned through gradient descent during online interactions with the environment. We conduct experiments using multiple RL environments and the experimental results demonstrate that our method outperforms several classic and state-of-the-art comparison methods in terms of overall safe RL performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。