将目标导向强化学习转化为生存学习，提升稳定性和样本效率。
SVL: Goal-Conditioned Reinforcement Learning as Survival Learning
- 用生存概率建模状态到目标的时长，避免传统TD方法的不稳定
- 在离线基准上表现优于主流算法，尤其擅长长周期任务
- 适合复杂、长时序的机器人控制场景
标准的目标导向强化学习（GCRL）依赖时序差分学习，易因自举（bootstrapping）导致不稳定性与样本效率低。现有工作尝试对比学习和监督学习提升稳定性，本文提出一种概率化替代方案——生存价值学习（SVL），将GCRL重构为生存学习问题，通过概率分布建模每个状态到达目标的时间。该结构化分布蒙特卡洛视角导出闭式恒等式，将目标条件价值函数表示为生存概率的折扣和，支持通过最大似然法在事件发生和右删失轨迹上训练风险模型进行价值估计。本文设计三种实用价值估计器，包括有限时域截断及两种分箱无限时域近似，以捕捉长时程目标。在离线GCRL基准测试中，结合层次化执行器的SVL表现匹配或超越强基线，尤其在复杂长时任务中优势显著。
原文摘要 · Abstract (English)
Standard approaches to goal-conditioned reinforcement learning (GCRL) that rely on temporal-difference learning can be unstable and sample-inefficient due to bootstrapping. While recent work has explored contrastive and supervised formulations to improve stability, we present a probabilistic alternative, called survival value learning (SVL), that reframes GCRL as a survival learning problem by modeling the time-to-goal from each state as a probability distribution. This structured distributional Monte Carlo perspective yields a closed-form identity that expresses the goal-conditioned value function as a discounted sum of survival probabilities, enabling value estimation via a hazard model trained via maximum likelihood on both event and right-censored trajectories. We introduce three practical value estimators, including finite-horizon truncation and two binned infinite-horizon approximations to capture long-horizon objectives. Experiments on offline GCRL benchmarks show that SVL combined with hierarchical actors matches or surpasses strong hierarchical TD and Monte Carlo baselines, excelling on complex, long-horizon tasks. Webpage and Code: https://simple-robotics.github.io/publications/survival-value-learning/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。