让强化学习在约束下更稳定高效，不牺牲最终回报。
Central Path Proximal Policy Optimization
- 用中央路径引导策略更新，保持优化轨迹贴近约束最优解
- 相比现有方法，约束满足更严格，性能提升明显
- 适合需要强约束的现实场景，如机器人控制与安全决策
在受限马尔可夫决策过程（CMDP）中，传统做法常导致最终回报下降。近期研究发现，将约束直接融入策略几何，可使优化轨迹靠近障碍法的中央路径，从而不影响最终回报。本文在此基础上提出中央路径近端策略优化（C3PO），对PPO损失进行简单修改，使策略迭代始终贴近约束优化问题的中央路径。相较于现有在线策略方法，C3PO在保证更紧约束的前提下实现更高性能，表明基于中央路径的更新是受限策略优化的有前景方向。
原文摘要 · Abstract (English)
In constrained Markov decision processes, enforcing constraints during training is often thought of as decreasing the final return. Recently, it was shown that constraints can be incorporated directly into the policy geometry, yielding an optimization trajectory close to the central path of a barrier method, which does not compromise final return. Building on this idea, we introduce Central Path Proximal Policy Optimization (C3PO), a simple modification of the PPO loss that produces policy iterates, that stay close to the central path of the constrained optimization problem. Compared to existing on-policy methods, C3PO delivers improved performance with tighter constraint enforcement, suggesting that central path-guided updates offer a promising direction for constrained policy optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。