通过分离视觉特征提升强化学习泛化能力,避免无关信息干扰。
Focus On What Matters: Separated Models For Visual-Based RL Generalization
- 分两路提取任务相关与无关特征,联合重建增强表征
- 在DMC视频背景任务中达到最优泛化性能
- 适合需要鲁棒视觉决策的机器人应用
视觉强化学习的核心挑战在于跨未见环境的有效泛化。尽管已有研究尝试多种辅助任务以提升泛化性,但因图像重建可能加剧对任务无关特征的过拟合,较少被采用。鉴于图像重建在表征学习中的优势,我们提出SMG(Separated Models for Generalization),一种利用图像重建促进泛化的新型方法。SMG通过双分支结构从视觉观测中分别提取任务相关与任务无关表征,并通过协同重建实现分离。在此基础上,进一步强调任务相关特征对泛化的重要性,引入两个一致性损失,引导智能体在不同场景下聚焦任务相关区域,从而有效避免过拟合。在DMC上的大量实验表明,SMG在泛化性能上达到当前最优,尤其在视频背景设置下表现突出。机器人操作任务的评估进一步验证了SMG在真实场景中的鲁棒性。
原文摘要 · Abstract (English)
A primary challenge for visual-based Reinforcement Learning (RL) is to generalize effectively across unseen environments. Although previous studies have explored different auxiliary tasks to enhance generalization, few adopt image reconstruction due to concerns about exacerbating overfitting to task-irrelevant features during training. Perceiving the pre-eminence of image reconstruction in representation learning, we propose SMG (Separated Models for Generalization), a novel approach that exploits image reconstruction for generalization. SMG introduces two model branches to extract task-relevant and task-irrelevant representations separately from visual observations via cooperatively reconstruction. Built upon this architecture, we further emphasize the importance of task-relevant features for generalization. Specifically, SMG incorporates two additional consistency losses to guide the agent's focus toward task-relevant areas across different scenarios, thereby achieving free from overfitting. Extensive experiments in DMC demonstrate the SOTA performance of SMG in generalization, particularly excelling in video-background settings. Evaluations on robotic manipulation tasks further confirm the robustness of SMG in real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。