用波动率机制提升强化学习安全性,兼顾收益与风险控制。
BCPPO: Bachelier-Inspired Constrained Proximal Policy Optimization for Tail-Risk-Aware Safe Reinforcement Learning

- 通过多个独立训练的代价网络分歧生成平滑惩罚项
- 在175次实验中同时优于所有对比方法的回报与尾部风险
- 适合需要安全部署且仅保留策略网络的应用场景
期望成本约束仍可能允许罕见的高成本事件。蒙特卡洛条件风险价值(CVaR)梯度在高置信度下噪声较大,而建模结果分布的评估网络会增加复杂性。本文提出BCPPO(Bachelier-inspired Constrained Proximal Policy Optimization),一种近端策略优化(PPO)方法。采用独立初始化的代价预测网络(评估器),通过随机样本掩码训练,其预测分歧反映了对训练数据中状态-动作区域及评估器训练的敏感性。利用巴舍利耶公式将该分歧转化为平滑的策略更新惩罚项。该惩罚项梯度不改变评估器,因此时序差分(TD)评估器学习保持不变。饱和感知控制器调整均值成本惩罚,防止剪裁期间累积误差增长。部署时仅保留策略网络。该分歧惩罚既非尾部事件概率,也非保证误差界,不提供安全保证。在175次共享任务、成本预算、训练步数和评估种子的实验中,无任何比较方法在任一任务中同时获得更高平均回报和更低平均CVaR。在Push1任务中,BCPPO的回报不低于所有对比方法,且在至少一项上实现严格提升。结果支持了在奖励、对评估分歧的谨慎处理以及仅策略部署之间实现实用平衡。
原文摘要 · Abstract (English)
Expected-cost constraints can still permit rare, high-cost events. Monte Carlo conditional value at risk (CVaR) gradients can be noisy at high confidence, whereas critics that model an outcome distribution add complexity. We propose BCPPO (Bachelier-Inspired Constrained Proximal Policy Optimization), a proximal policy optimization (PPO) method. Separately initialized cost-prediction networks (critics), trained with random sample masks, produce disagreement that marks predictions sensitive to which state-action regions occur in the training data and to critic training. A Bachelier formula for the expected amount above a reference level converts this disagreement into a smooth policy-update penalty. Gradients from this penalty do not alter the critics, so temporal-difference (TD) critic learning is unchanged. A saturation-aware controller adjusts the mean-cost penalty and stops accumulated error from growing while that penalty is clipped. Deployment retains only the policy network. The disagreement penalty is neither a tail-event probability nor a guaranteed error bound, and it provides no safety guarantee. Across 175 runs with shared tasks, costs, budgets, training steps, and evaluation seeds, no comparator attains both higher mean return and lower mean CVaR than BCPPO in any task. On Push1, BCPPO has no lower return and no higher CVaR than every comparator, with at least one strict gain. These results support a practical balance among reward, caution around cost predictions that vary across trained critics, and policy-only deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。