提升强化学习对难达目标的泛化能力，让智能体更懂如何抵达复杂目标。
ViSA: Visited-State Augmentation for Generalized Goal-Space Contrastive Reinforcement Learning
- 用访问过的状态生成新样本，增强探索困难区域的能力。
- 在模拟和真实机器人任务中，对难达目标的价值估计准确率显著提升。
- 适合研究目标导向强化学习与机器人控制的学者参考。
目标条件强化学习（GCRL）旨在学习一个能到达任意给定目标的策略。对比强化学习（CRL）通过对比学习近似价值函数来更新策略，相比传统方法具有更高的采样效率。然而，由于CRL在学习过程中将访问过的状态视为伪目标，仅能对有限目标进行准确的价值函数估计。为此，我们提出一种名为ViSA（Visited-State Augmentation）的新数据增强方法。ViSA包含两个部分：1）生成增强的状态样本，以在在线探索中增强难以访问状态的样本；2）学习一致的嵌入空间，通过重构嵌入空间的目标函数，利用增强状态作为辅助信息，基于互信息进行正则化。我们在仿真和真实机器人任务中评估了ViSA，结果表明其提升了目标空间的泛化能力，使对难以访问目标的价值估计更加准确。更多细节请见项目主页：https://issa-n.github.io/projectPage_ViSA/
原文摘要 · Abstract (English)
Goal-Conditioned Reinforcement Learning (GCRL) is a framework for learning a policy that can reach arbitrarily given goals. In particular, Contrastive Reinforcement Learning (CRL) provides a framework for policy updates using an approximation of the value function estimated via contrastive learning, achieving higher sample efficiency compared to conventional methods. However, since CRL treats the visited state as a pseudo-goal during learning, it can accurately estimate the value function only for limited goals. To address this issue, we propose a novel data augmentation approach for CRL called ViSA (Visited-State Augmentation). ViSA consists of two components: 1) generating augmented state samples, with the aim of augmenting hard-to-visit state samples during on-policy exploration, and 2) learning consistent embedding space, which uses an augmented state as auxiliary information to regularize the embedding space by reformulating the objective function of the embedding space based on mutual information. We evaluate ViSA in simulation and real-world robotic tasks and show improved goal-space generalization, which permits accurate value estimation for hard-to-visit goals. Further details can be found on the project page: https://issa-n.github.io/projectPage_ViSA/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。