arXiv:2502.08336cs.AI2025-02被引 1

让智能体无视干扰像素,学会在新场景中稳定执行任务。

Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learning

  • 用显著性引导的值一致性模块,专注关键视觉信息。
  • 在三大基准上平均提升39%以上,尤其在复杂场景下表现突出。
  • 适合需要强泛化能力的视觉强化学习应用。

视觉强化学习中的策略泛化仍面临重大挑战,因智能体易对训练环境的特定视觉观测过拟合。在未见过的环境中,干扰像素可能导致智能体提取与任务无关的信息,从而偏离训练时学得的最优行为,阻碍泛化。为此,我们提出显著性不变一致策略学习(SCPL)算法,一种高效的零样本泛化框架。该方法引入新的值一致性模块与动态性模块,有效捕捉任务相关表征。值一致性模块通过显著性引导,确保智能体在原始和扰动观测中均聚焦于任务相关像素;动态性模块利用增强数据帮助编码器捕获动态与奖励相关表征。此外,理论分析强调策略一致性对泛化的重要性,因此引入带有KL散度约束的策略一致性模块,保持原始与扰动观测间策略的一致性。在DMC-GB、Robotic Manipulation和CARLA基准上的大量实验表明,SCPL显著优于现有先进方法。特别地,在具有挑战性的DMC视频难设置、机器人难设置和CARLA基准上,分别实现14%、39%和69%的平均性能提升。

原文摘要 · Abstract (English)

Generalizing policies to unseen scenarios remains a critical challenge in visual reinforcement learning, where agents often overfit to the specific visual observations of the training environment. In unseen environments, distracting pixels may lead agents to extract representations containing task-irrelevant information. As a result, agents may deviate from the optimal behaviors learned during training, thereby hindering visual generalization.To address this issue, we propose the Salience-Invariant Consistent Policy Learning (SCPL) algorithm, an efficient framework for zero-shot generalization. Our approach introduces a novel value consistency module alongside a dynamics module to effectively capture task-relevant representations. The value consistency module, guided by saliency, ensures the agent focuses on task-relevant pixels in both original and perturbed observations, while the dynamics module uses augmented data to help the encoder capture dynamic- and reward-relevant representations. Additionally, our theoretical analysis highlights the importance of policy consistency for generalization. To strengthen this, we introduce a policy consistency module with a KL divergence constraint to maintain consistent policies across original and perturbed observations.Extensive experiments on the DMC-GB, Robotic Manipulation, and CARLA benchmarks demonstrate that SCPL significantly outperforms state-of-the-art methods in terms of generalization. Notably, SCPL achieves average performance improvements of 14\%, 39\%, and 69\% in the challenging DMC video hard setting, the Robotic hard setting, and the CARLA benchmark, respectively.Project Page: https://sites.google.com/view/scpl-rl.

强化学习视觉泛化策略一致性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。