提出安全对比强化学习,用失败信号提升机器人避障与目标达成能力。
Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals

- 基于失败信号修正对比学习中的过估计偏差,引入质量加权与生存概率补偿
- 在12个高风险任务中,存活率显著提升,目标达成性能超越基线方法
- 仅需失败与否的二值信号,适合复杂物理环境下的安全策略训练
对比强化学习(CRL)在目标导向任务中表现良好,但在失败终止的马尔可夫决策过程中,传统方法仅考虑失败前的未来目标作为正样本,忽略了失败导致的概率质量损失。理论分析表明,这种忽略会引发目标达成价值的系统性过估计,使接近失败的轨迹获得过强监督信号,反而强化危险行为。为此,本文提出两种轻量但有效的修正:质量加权InfoNCE修正评价网络中的短存续未来权重过高问题,对数生存质量得分恢复策略优化中缺失的生存概率信息。由此构建的安全对比强化学习(Safe-CRL)仅需失败终止的二值信号,即可实现安全的目标条件策略学习。在12个高失败风险的机器人导航与运动任务中,Safe-CRL一致提升存活率,并显著优于Scaling-CRL基线,在目标达成性能上取得明显进步。深层安全策略还展现出复杂的避险行为。本研究完善了失败终止下的CRL理论,提供了一种可扩展的安全强化学习框架。代码已开源。
原文摘要 · Abstract (English)
Contrastive reinforcement learning (CRL) scales effectively in goal-conditioned tasks by casting policy learning into a self-supervised contrastive objective. However, in a failure-terminated Markov decision process, established CRL considers pre-failure future goals only when constructing positive samples, without accounting for the probability mass removed by failure termination. Our theoretical analysis shows that this omission induces a systematic overestimation bias in goal-reaching values. Consequently, near-failure trajectories provide disproportionately strong supervision of success despite retaining little future occupancy. Unsafe actions can thereby be reinforced through catastrophic failure bootstrapping, leading to failed policy learning and unsustainable goal-reaching behaviours. To address this problem, we introduce two minimal yet strong corrections: mass-weighted InfoNCE corrects the overweighting of short surviving futures in critic learning, and a log-survival-mass score restores the missing survival mass in policy optimization. The resulting method, Safe Contrastive Reinforcement Learning (Safe-CRL), requires only the one-bit signal provided by failure termination to scale safe goal-conditioned policy learning. Across twelve failure-prone robot navigation and locomotion tasks, Safe-CRL consistently improves survival and substantially outperforms the Scaling-CRL baseline in goal-reaching performance. Additionally, deep Safe-CRL policies exhibit complex failure-avoidance behaviours. This study completes the CRL theory under failure termination and provides a scalable safe RL framework. The code is available via https://github.com/RomainLITUD/safe-crl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。