arXiv:2609.08642cs.LGcs.AI2026-09

提出新探索机制,让智能体同时追求新颖和可达成的目标。

SUN: Reaching for Novelty in Reinforcement Learning

论文配图:SUN: Reaching for Novelty in Reinforcement Learning
图 1 · 摘自论文原文
  • 用继承值函数构造新颖性与可达性联合指标SUN
  • 在多种复杂环境中性能超越现有方法,尤其擅长处理不可达目标
  • 轻量级伪计数设计,适合实际部署

强化学习中的探索仍是根本挑战。近期基于目标的强化学习策略虽能拓展状态覆盖,但未能同时考虑目标的新颖性与可达性:二者常被手动权衡、顺序使用或忽略其一。本文提出一种兼顾可达性的目标选择框架,可无缝集成于任意离策略强化学习算法。为此,我们引入基于继承值函数的SUccessor-to-Novelty(SUN)指标,用于识别既新颖又可达的目标。理论上,SUN在极限下等价于计数奖励,可界定短时命中概率,并严格排除不可达目标。进一步设计了自适应目标选择策略及高效轻量的伪计数机制,避免传统方法的计算开销。实验验证表明,SUN在标准与新型环境(含不可达状态、难达区域、不可逆转移、障碍物、迷宫、无界空间)中持续优于当前最优方法。

原文摘要 · Abstract (English)

Exploration in reinforcement learning (RL) remains a fundamental challenge. Recent goal-conditioned RL strategies (which select goals to encourage broader state coverage) have shown promising results, but none scores a goal by novelty and reachability jointly: the two signals are traded off by hand, applied in sequence, or one is neglected outright. In this paper, we introduce a reachability-aware goal-selection framework that explicitly integrates these two aspects, and that can be seamlessly incorporated into any off-policy RL algorithm. To this aim, we propose SUccessor-to-Novelty (SUN), an indicator derived from successor value functions to identify goals that are both novel and reachable. We prove that SUN recovers count-based bonuses in the limit, bounds short-horizon hitting probabilities, and provably rejects unreachable goals. We further present an adaptive goal-selection strategy that leverages these properties, and an accurate yet lightweight pseudocount to avoid the overhead of classic methods. We back up all our claims with thorough benchmarks: SUN consistently outperforms state-of-the-art methods in standard and novel environments with unreachable or hard-to-reach states, irreversible transitions, obstacles, mazes, and unbounded spaces.

强化学习探索机制目标导向新颖性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。