让强化学习更快更稳地遵守安全规则，避免反复出错。
CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
- 用约束敏感度指导策略更新，提前预判安全边界
- 实验显示恢复安全速度更快，奖励损失更少
- 适合需要高安全性的机器人控制场景
安全强化学习旨在最大化期望回报的同时满足安全约束，通常建模为约束马尔可夫决策过程（CMDP）。尽管原始-对偶方法在深度强化学习中可扩展，但常因约束修正延迟导致振荡行为和长期安全违规。本文提出约束敏感策略优化（CSPO），一种一阶原始-对偶方法，将局部约束敏感度融入策略更新。CSPO 在原始目标中加入基于到安全边界最短有符号距离的约束敏感修正项，实现更智能的安全恢复，补偿拉格朗日乘子更新延迟，减少边界附近的振荡，并保持原约束问题的KKT解。在导航与运动基准测试上的实验表明，相比最先进的原始-对偶与惩罚法，CSPO实现了更快的安全恢复和更高的奖励保留，从而获得更高的约束回报。
原文摘要 · Abstract (English)
Safe reinforcement learning (Safe RL) aims to maximize expected return while satisfying safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods scale well to deep RL, they often suffer from delayed constraint correction, leading to oscillatory behavior and prolonged safety violations. In this paper, we propose Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method that incorporates local constraint sensitivity into policy updates. CSPO augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary, enabling smarter recovery steps back to safety, compensating for delayed Lagrange multiplier updates, reducing oscillations near the boundary, and preserving the KKT solutions of the original constrained problem. Experiments on navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and high reward preservation, resulting in higher constrained returns compared to state-of-the-art primal-dual and penalty-based methods
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。