提出安全强化学习新算法，兼顾零成本安全与任务效率。
SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints
- 动态组合奖励与成本梯度，分步优化安全与性能。
- 实验显示在安全环境任务中实现最优安全与表现平衡。
- 适合自动驾驶、医疗等高风险场景的强化学习应用。
在安全关键领域，强化学习代理必须满足严格的零成本安全约束以完成任务。现有无模型方法往往无法实现近零安全违规，或变得过于保守。本文提出安全偏置信任区域策略优化（SB-TRPO），一种针对硬约束强化学习的原理性算法，可动态平衡成本降低与奖励提升。每一步更新中，SB-TRPO 通过奖励与成本自然策略梯度的动态凸组合进行优化，确保固定比例的最优成本降低，同时利用剩余更新能力提升奖励。该方法具有安全性的局部进展形式保证，且在梯度方向匹配时仍能持续改进奖励。在标准及挑战性 Safety Gymnasium 任务上的实验表明，SB-TRPO 在硬约束环境下始终实现最佳的安全性与任务性能平衡。
原文摘要 · Abstract (English)
In safety-critical domains, reinforcement learning (RL) agents must often satisfy strict, zero-cost safety constraints while accomplishing tasks. Existing model-free methods frequently either fail to achieve near-zero safety violations or become overly conservative. We introduce Safety-Biased Trust Region Policy Optimisation (SB-TRPO), a principled algorithm for hard-constrained RL that dynamically balances cost reduction with reward improvement. At each step, SB-TRPO updates via a dynamic convex combination of the reward and cost natural policy gradients, ensuring a fixed fraction of optimal cost reduction while using remaining update capacity for reward improvement. Our method comes with formal guarantees of local progress on safety, while still improving reward whenever gradients are suitably aligned. Experiments on standard and challenging Safety Gymnasium tasks demonstrate that SB-TRPO consistently achieves the best balance of safety and task performance in the hard-constrained regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。