ACPO通过动态调整预算，平衡奖励与约束，提升安全强化学习性能。
Adversarial Constrained Policy Optimization: Improving Constrained Reinforcement Learning by Adapting Budgets
- 将约束问题拆分为对抗性两阶段交替优化
- 在Safety Gymnasium和四足机器人任务中优于主流基线
- 适合需兼顾性能与安全的强化学习应用
约束强化学习在安全关键领域取得进展，但难以平衡任务表现与约束满足，易陷入过度保守或违反约束的局部最优。本文提出对抗性约束策略优化（ACPO），在训练中同时优化奖励并自适应调整成本预算。方法将原问题分解为两个交替求解的对抗阶段，理论保证策略更新性能。在Safety Gymnasium和四足运动任务上的实验表明，该算法性能优于常用基线。
原文摘要 · Abstract (English)
Constrained reinforcement learning has achieved promising progress in safety-critical fields where both rewards and constraints are considered. However, constrained reinforcement learning methods face challenges in striking the right balance between task performance and constraint satisfaction and it is prone for them to get stuck in over-conservative or constraint violating local minima. In this paper, we propose Adversarial Constrained Policy Optimization (ACPO), which enables simultaneous optimization of reward and the adaptation of cost budgets during training. Our approach divides original constrained problem into two adversarial stages that are solved alternately, and the policy update performance of our algorithm can be theoretically guaranteed. We validate our method through experiments conducted on Safety Gymnasium and quadruped locomotion tasks. Results demonstrate that our algorithm achieves better performances compared to commonly used baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。