将安全预算动态转化为实时风险阈值,提升非平稳强化学习的决策安全性。
From Cumulative Constraints to Adaptive Runtime Safety Control for Nonstationary Reinforcement Learning
- 通过在线投影安全预算生成可变风险阈值,实现动态安全控制。
- 在高速公路汇入场景中,安全违规减少,干预更精准,性能损失可控。
- 适合需要持续安全保障的自动驾驶等实时系统应用。
强化学习中的安全通常以累积成本约束形式定义,但这类轨迹级保障无法直接防止单个不安全决策,尤其在非平稳环境中更为突出。由于相同动作的风险随上下文变化,固定状态级阈值可能过于保守或过弱。本文提出约束投影安全盾(CPSS),一种运行时机制,将累积安全预算转化为执行过程中的自适应状态级控制约束。CPSS 实时跟踪剩余安全预算,将其投影为时变可接受风险阈值,并过滤预测安全成本超过当前阈值的策略动作。阈值基于上下文信号在线调整,在高挑战或快速变化环境中收紧,在预算充足时放宽。理论分析表明,该机制保证每一步动作满足状态级阈值,诱导有限时域累积成本上界,并给出性能退化与干预频率及单步奖励畸变的关系。在 highway-env 的非平稳高速汇入场景中,多种子实验显示,CPSS 显著降低基于距离的安全违规,增大车距,同时选择性干预而非主导学习策略。结果支持将累积安全规范通过预算到阈值的自适应映射,实现连续强化学习系统的有效局部安全控制。
原文摘要 · Abstract (English)
Safety in reinforcement learning is often specified through cumulative cost constraints, but these trajectory-level guarantees do not directly prevent unsafe individual decisions, especially under nonstationarity. In continual and nonstationary settings, the difficulty is amplified because the risk associated with the same action can vary across contexts, while a fixed state-level threshold may be either too conservative or too weak. We propose Constraint Projection Safety Shield (CPSS), a runtime mechanism that converts a cumulative safety budget into adaptive state-level control constraints during execution. CPSS tracks the remaining safety budget, projects it into a time-varying admissible risk threshold, and filters policy actions whose predicted safety cost exceeds the active threshold. The threshold is adjusted online using contextual signals so that enforcement becomes stricter in more demanding or rapidly changing regimes and less restrictive when the available safety budget is sufficient. We analyze the resulting shielded policy and show that the mechanism guarantees per-state threshold satisfaction for executed actions, induces finite-horizon cumulative cost bounds, and yields a performance degradation bound in terms of intervention frequency and per-step reward distortion. We evaluate CPSS in nonstationary highway merging scenarios using highway-env. Across multiple seeds, CPSS substantially reduces proximity-based safety violations and increases separation margins while intervening selectively rather than dominating the learned policy. These results support adaptive budget-to-threshold projection as a practical way to transform cumulative safety specifications into effective local safety control for continual reinforcement learning systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。