让强化学习始终安全：通过重构策略空间确保训练不越界
Embedding Safety into RL: A New Take on Trust Region Methods
- 重构策略空间几何，使信任区域只包含安全策略
- 训练全程零约束违规，同时保持与最优回报相当的性能
- 适合对安全性要求高的机器人、自动驾驶等场景
强化学习代理虽能解决多种任务，但常表现出不安全行为。受限马尔可夫决策过程（CMDPs）通过施加安全约束来应对这一问题，但现有方法要么牺牲奖励最大化，要么允许不安全训练。本文提出受限信任区域策略优化（C-TRPO），通过重塑策略空间几何，确保信任区域仅包含安全策略，从而在训练全程保证约束满足。我们分析了其理论性质，并揭示其与TRPO、自然策略梯度（NPG）及受限策略优化（CPO）的联系。实验表明，C-TRPO在维持有竞争力回报的同时显著减少约束违反。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) agents can solve diverse tasks but often exhibit unsafe behavior. Constrained Markov Decision Processes (CMDPs) address this by enforcing safety constraints, yet existing methods either sacrifice reward maximization or allow unsafe training. We introduce Constrained Trust Region Policy Optimization (C-TRPO), which reshapes the policy space geometry to ensure trust regions contain only safe policies, guaranteeing constraint satisfaction throughout training. We analyze its theoretical properties and connections to TRPO, Natural Policy Gradient (NPG), and Constrained Policy Optimization (CPO). Experiments show that C-TRPO reduces constraint violations while maintaining competitive returns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。