用对比学习提升安全强化学习中的探索效率。
Enhance Exploration in Safe Reinforcement Learning with Contrastive Representation Learning
- 通过自编码器与对比学习构建状态表征，区分安全与危险状态。
- 在稀疏奖励环境中探索更充分，同时保持高安全性和效率。
- 适合需要安全探索的机器人导航等实际场景使用。
在安全强化学习中，智能体需在探索行为与安全约束间取得平衡。现有基于领域迁移的方法通过相关环境学习先验Q函数以避免不安全动作，但因存在大量误报，部分安全动作从未被执行，导致稀疏奖励环境中探索不足。本文旨在学习高效的状态表征，在稀疏奖励环境中实现探索与安全偏好动作的平衡。首先，利用自编码器将图像输入映射为潜在表示；进一步引入对比学习目标，以区分安全与不安全状态。在训练阶段，使用潜在空间距离构建额外安全检查机制，使智能体在访问不安全状态时能主动调整探索方向。实验在三个基于导航的MiniGrid环境上进行，结果表明，该方法能在保证安全的前提下显著提升探索能力。
原文摘要 · Abstract (English)
In safe reinforcement learning, agent needs to balance between exploration actions and safety constraints. Following this paradigm, domain transfer approaches learn a prior Q-function from the related environments to prevent unsafe actions. However, because of the large number of false positives, some safe actions are never executed, leading to inadequate exploration in sparse-reward environments. In this work, we aim to learn an efficient state representation to balance the exploration and safety-prefer action in a sparse-reward environment. Firstly, the image input is mapped to latent representation by an auto-encoder. A further contrastive learning objective is employed to distinguish safe and unsafe states. In the learning phase, the latent distance is used to construct an additional safety check, which allows the agent to bias the exploration if it visits an unsafe state. To verify the effectiveness of our method, the experiment is carried out in three navigation-based MiniGrid environments. The result highlights that our method can explore the environment better while maintaining a good balance between safety and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。