动态环境变化时自动调整安全防护，保障强化学习不越界。
Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shifts
- 在线推断隐藏参数的低维表示，动态调整安全屏障。
- 在多个基准测试中，兼顾收益与安全表现优于基线方法。
- 适合对安全性要求高、环境参数可能突变的智能系统应用。
未知的环境动态变化（如摩擦、重力等隐藏参数）给强化学习的安全性带来挑战。本文提出自适应屏蔽（Adaptive Shielding）框架，用于受限隐藏参数马尔可夫决策过程中的安全强化学习。通过函数编码器从转移数据中在线推断底层动态的低维表示，使安全屏障能够自适应调整。为确保调整过程中的安全性，采用双层策略：首先使用安全正则化优化，主动引导策略避开高成本区域；其次，自适应屏蔽基于推断出的动态进行风险预测，并利用置信区间方法（符合性预测）施加不确定性感知的约束以过滤不安全动作。理论证明了屏蔽中的预测误差与平均成本率上界相关。实验表明，在包含不同隐藏参数的Safe-Gym基准上，该方法在收益-安全权衡上优于基线，且能可靠泛化至未见过的动态，仅带来轻微执行开销。
原文摘要 · Abstract (English)
Unseen shifts in environment dynamics, driven by hidden parameters such as friction or gravity, create a challenge for maintaining safety. We address this challenge by proposing Adaptive Shielding, a framework for safe reinforcement learning in constrained hidden-parameter Markov decision processes. A function encoder infers a low-dimensional representation of the underlying dynamics online from transition data, allowing the shield to adapt. To ensure safety during this process, we use a two-layer strategy. First, we introduce safety-regularized optimization that proactively trains the policy away from high-cost regions. Second, the adaptive shielding reactively uses the inferred dynamics to forecast safety risks and applies uncertainty-aware bounds using conformal prediction to filter unsafe actions. We prove that prediction errors in the shielding connect with bounds on the average cost rate. Empirically, across Safe-Gym benchmarks with varying hidden parameters, our approach outperforms baselines on the return-safety trade-off and generalizes reliably to unseen dynamics, while incurring only modest execution-time overhead. Code is available at https://github.com/safe-autonomy-lab/AdaptiveShieldingFE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。