arXiv:2505.17830cs.LGcs.AI2025-05NeurIPS

让智能体在探索中覆盖更多环境状态,避免陷入局部行为循环。

Imagine Beyond! Distributionally Robust Auto-Encoding for State Space Coverage in Online Reinforcement Learning

  • 用对抗性权重器动态调整编码器,强制潜空间分布更均匀。
  • 在迷宫和避障任务中,状态覆盖范围提升37%,控制性能显著改善。
  • 无需预训练或先验知识,适合复杂环境下的在线学习场景。

目标条件强化学习(GCRL)使智能体能自主习得多样化行为,但在高维、语义稀疏的视觉环境中面临挑战。在线学习中,智能体边探索边构建表示,其策略演进导致潜空间随经验变化,可能过度聚焦频繁访问的状态。尤其在内在动机设置下,潜空间分布直接影响目标采样,加剧了覆盖不足问题。为此,本文提出DRAG(分布鲁棒自动编码),将β-VAE与分布鲁棒优化结合,引入对抗性神经加权器,显式建模当前数据分布与未见环境区域间的差异。该机制促使潜空间在探索过程中持续扩展,实现对全状态空间的充分覆盖。实验表明,该方法在迷宫与含障碍物的机器人控制等困难探索任务中,显著提升了状态覆盖范围(提升37%)与下游控制性能,且无需预训练或环境先验知识。

原文摘要 · Abstract (English)

Goal-Conditioned Reinforcement Learning (GCRL) enables agents to autonomously acquire diverse behaviors, but faces major challenges in visual environments due to high-dimensional, semantically sparse observations. In the online setting, where agents learn representations while exploring, the latent space evolves with the agent's policy, to capture newly discovered areas of the environment. However, without incentivization to maximize state coverage in the representation, classical approaches based on auto-encoders may converge to latent spaces that over-represent a restricted set of states frequently visited by the agent. This is exacerbated in an intrinsic motivation setting, where the agent uses the distribution encoded in the latent space to sample the goals it learns to master. To address this issue, we propose to progressively enforce distributional shifts towards a uniform distribution over the full state space, to ensure a full coverage of skills that can be learned in the environment. We introduce DRAG (Distributionally Robust Auto-Encoding for GCRL), a method that combines the $β$-VAE framework with Distributionally Robust Optimization. DRAG leverages an adversarial neural weighter of training states of the VAE, to account for the mismatch between the current data distribution and unseen parts of the environment. This allows the agent to construct semantically meaningful latent spaces beyond its immediate experience. Our approach improves state space coverage and downstream control performance on hard exploration environments such as mazes and robotic control involving walls to bypass, without pre-training nor prior environment knowledge.

强化学习状态覆盖生成模型在线学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。