arXiv:2410.00704cs.LGcs.AI2024-10被引 5

通过对比学习与霍普菲尔德网络,自动提炼强化学习中的抽象状态。

Contrastive Abstraction for Reinforcement Learning

  • 基于相邻状态应属同一抽象态的假设,用对比学习聚类状态表示。
  • 利用现代霍普菲尔德网络将相似表示映射到固定点,形成可调抽象层级。
  • 无需奖励信号,适用于多种下游强化学习任务,提升训练效率。

在涉及大量状态的长轨迹中,强化学习面临挑战。为有效解决此问题,可通过聚类状态来减少状态数量。尽管深度强化学习能发现抽象状态,但端到端学习不稳定。本文提出对比抽象学习:假设轨迹中连续状态属于同一抽象状态(如位置、子目标、库存或生命值)。该方法分为两阶段:第一阶段为自监督对比学习,强制相邻状态具有相似表示;第二阶段使用现代霍普菲尔德网络,将相似表示映射至相同固定点,即抽象状态。抽象程度可通过设定霍普菲尔德网络的固定点数量调节。此外,该方法不依赖奖励信号,可显著提升多种下游强化学习任务的效率。实验验证了其有效性。

原文摘要 · Abstract (English)

Learning agents with reinforcement learning is difficult when dealing with long trajectories that involve a large number of states. To address these learning problems effectively, the number of states can be reduced by abstract representations that cluster states. In principle, deep reinforcement learning can find abstract states, but end-to-end learning is unstable. We propose contrastive abstraction learning to find abstract states, where we assume that successive states in a trajectory belong to the same abstract state. Such abstract states may be basic locations, achieved subgoals, inventory, or health conditions. Contrastive abstraction learning first constructs clusters of state representations by contrastive learning and then applies modern Hopfield networks to determine the abstract states. The first phase of contrastive abstraction learning is self-supervised learning, where contrastive learning forces states with sequential proximity to have similar representations. The second phase uses modern Hopfield networks to map similar state representations to the same fixed point, i.e.\ to an abstract state. The level of abstraction can be adjusted by determining the number of fixed points of the modern Hopfield network. Furthermore, \textit{contrastive abstraction learning} does not require rewards and facilitates efficient reinforcement learning for a wide range of downstream tasks. Our experiments demonstrate the effectiveness of contrastive abstraction learning for reinforcement learning.

强化学习抽象表示对比学习霍普菲尔德网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。