通过分布风险控制与参数空间探索，提升强化学习的安全性与泛化能力。
Safe Langevin Soft Actor Critic
- 用自适应Langevin动态增强价值函数多样性，跳出局部最优。
- 结合CVaR与隐式分位数网络，有效缓解极端风险事件影响。
- 动态调整约束惩罚，适合对安全性要求高的机器人控制任务。
在受限强化学习中，奖励与安全的平衡因尖锐值最小点导致的泛化差以及重尾风险分布处理不足而难以实现。我们提出安全朗之万软演员-评论家（SL-SAC），通过参数空间探索和分布风险控制，同时解决上述问题。该方法包含三个核心机制：(1) 采用自适应随机梯度朗之万动力学（aSGLD）优化奖励评论家，提升集成多样性并摆脱不良极小值；(2) 利用隐式分位数网络（IQN）进行分布式成本估计，并通过条件风险价值（CVaR）优化来抑制尾部风险；(3) 采用反应式拉格朗日松弛方案，根据经验CVaR动态调整约束执行力度。我们提供了CVaR估计误差的理论保证，并证明基于CVaR的拉格朗日更新比期望成本更新能产生更强的约束违反信号。在Safety-Gymnasium基准测试中，SL-SAC在10项任务中有7项达到最低成本，且保持了竞争力的回报，在速度类任务中成本降低19%-63%，优于现有最先进方法。
原文摘要 · Abstract (English)
Balancing reward and safety in constrained reinforcement learning remains challenging due to poor generalization from sharp value minima and inadequate handling of heavy-tailed risk distribution. We introduce Safe Langevin Soft Actor-Critic (SL-SAC), a principled algorithm that addresses both issues through parameter-space exploration and distributional risk control. Our approach combines three key mechanisms: (1) Adaptive Stochastic Gradient Langevin Dynamics (aSGLD) for reward critics, promoting ensemble diversity and escape from poor optima; (2) distributional cost estimation via Implicit Quantile Networks (IQN) with Conditional Value-at-Risk (CVaR) optimization for tail-risk mitigation; and (3) a reactive Lagrangian relaxation scheme that adapts constraint enforcement based on the empirical CVaR of episodic costs. We provide theoretical guarantees on CVaR estimation error and demonstrate that CVaR-based Lagrange updates yield stronger constraint violation signals than expected-cost updates. On Safety-Gymnasium benchmarks, SL-SAC achieves the lowest cost in 7 out of 10 tasks while maintaining competitive returns, with cost reductions of 19-63% in velocity tasks compared to state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。