通过可学习的注意力掩码提升强化学习策略在新任务上的泛化能力
Sparse Masked Attention Policies for Reliable Generalization
- 在注意力网络中引入可学习的掩码函数，动态过滤无关观测信息
- 在Procgen基准上显著优于标准PPO和传统掩码方法，新任务泛化性能提升明显
- 适合关注强化学习泛化性与模型鲁棒性的研究者和工程应用
在强化学习中，通过移除观测中的冗余信息来实现抽象的方法常用于提升策略在未见任务上的泛化能力。然而，这些方法常忽略一个关键弱点：用于提取低维表示的函数在未见观测下的泛化能力未知。本文提出一种更可靠泛化的信息移除方法，通过在基于注意力的策略网络中引入学习型掩码函数，该函数作用于注意力权重并与其深度融合。实验表明，该方法在Procgen基准上显著优于标准PPO及传统掩码方法，有效提升了策略在未见任务上的泛化性能。
原文摘要 · Abstract (English)
In reinforcement learning, abstraction methods that remove unnecessary information from the observation are commonly used to learn policies which generalize better to unseen tasks. However, these methods often overlook a crucial weakness: the function which extracts the reduced-information representation has unknown generalization ability in unseen observations. In this paper, we address this problem by presenting an information removal method which more reliably generalizes to new states. We accomplish this by using a learned masking function which operates on, and is integrated with, the attention weights within an attention-based policy network. We demonstrate that our method significantly improves policy generalization to unseen tasks in the Procgen benchmark compared to standard PPO and masking approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。