KFCPO让强化学习更安全高效，自动调节奖励与约束的平衡。
KFCPO: Kronecker-Factored Approximated Constrained Policy Optimization
- 用分层闭式近似加速策略优化，避免迭代开销。
- 在安全环境测试中，平均收益比基线高10.3%至50.2%。
- 适合需要高安全性与稳定性的实际强化学习场景。
我们提出KFCPO，一种新型安全强化学习算法，结合可扩展的克罗内克积近似曲率（K-FAC）二阶策略优化与安全感知梯度调控。KFCPO通过分层闭式方式近似费雪信息矩阵（FIM），实现高效稳定的自然梯度更新，避免迭代逼近的开销。为解决奖励最大化与约束满足之间的权衡，引入基于裕度感知的梯度调控机制，根据智能体靠近安全边界程度自适应调整奖励与代价梯度的影响。该方法采用方向敏感投影融合梯度，消除有害干扰，并避免固定硬阈值带来的突变。此外，采用小批量层级的KL回滚策略，确保信任区域合规，防止策略剧烈波动。在Safety Gymnasium上使用OmniSafe进行实验，KFCPO在各类环境中相比最优安全基线，平均回报提升10.3%至50.2%，展现出优异的安全性与性能平衡。
原文摘要 · Abstract (English)
We propose KFCPO, a novel Safe Reinforcement Learning (Safe RL) algorithm that combines scalable Kronecker-Factored Approximate Curvature (K-FAC) based second-order policy optimization with safety-aware gradient manipulation. KFCPO leverages K-FAC to perform efficient and stable natural gradient updates by approximating the Fisher Information Matrix (FIM) in a layerwise, closed form manner, avoiding iterative approximation overheads. To address the tradeoff between reward maximization and constraint satisfaction, we introduce a margin aware gradient manipulation mechanism that adaptively adjusts the influence of reward and cost gradients based on the agent's proximity to safety boundaries. This method blends gradients using a direction sensitive projection, eliminating harmful interference and avoiding abrupt changes caused by fixed hard thresholds. Additionally, a minibatch level KL rollback strategy is adopted to ensure trust region compliance and to prevent destabilizing policy shifts. Experiments on Safety Gymnasium using OmniSafe show that KFCPO achieves 10.3% to 50.2% higher average return across environments compared to the best baseline that respected the safety constraint, demonstrating superior balance of safety and performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。