arXiv:2502.20341cs.LG2025-02ICLR被引 3

学习状态相关的安全表征,让智能体更安全地探索。

Safety Representations for Safer Policy Learning

  • 用安全表征增强状态特征,引导更安全的探索
  • 训练中约束违反减少,任务表现显著提升
  • 适合高风险场景下的强化学习应用

强化学习算法通常需要对状态空间进行大量探索以寻找最优策略,但在安全关键应用中,这种探索可能带来灾难性后果。现有安全探索方法通过施加约束来缓解风险,但常导致行为过于保守、学习效率低下。早期约束违规会受到重罚,使智能体困在局部最优,回避高回报但高风险的状态区域。为此,我们提出一种显式学习状态条件安全表征的方法。通过将安全表征融入状态特征,该方法在不过度保守的前提下自然促进更安全的探索,从而在安全关键场景中实现更高效、更安全的策略学习。在多个环境中的实证评估表明,该方法显著提升了任务性能,同时减少了训练过程中的约束违反次数,验证了其在探索与安全间平衡的有效性。

原文摘要 · Abstract (English)

Reinforcement learning algorithms typically necessitate extensive exploration of the state space to find optimal policies. However, in safety-critical applications, the risks associated with such exploration can lead to catastrophic consequences. Existing safe exploration methods attempt to mitigate this by imposing constraints, which often result in overly conservative behaviours and inefficient learning. Heavy penalties for early constraint violations can trap agents in local optima, deterring exploration of risky yet high-reward regions of the state space. To address this, we introduce a method that explicitly learns state-conditioned safety representations. By augmenting the state features with these safety representations, our approach naturally encourages safer exploration without being excessively cautious, resulting in more efficient and safer policy learning in safety-critical scenarios. Empirical evaluations across diverse environments show that our method significantly improves task performance while reducing constraint violations during training, underscoring its effectiveness in balancing exploration with safety.

强化学习安全探索策略学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。