提出简单有效方法直接从像素学成功续特征,避免表示崩溃且训练更快。
Learning Successor Features the Simple Way
- 用TD损失和奖励预测损失联合训练,直接学习成功续特征
- 在2D/3D迷宫和Mujoco任务中性能媲美或超越现有方法
- 无需预训练,训练效率高,适合连续学习场景
深度强化学习中,如何在非平稳环境中学习不发生灾难性遗忘或干扰的表示仍具挑战。成功续特征(SFs)为此提供潜在解决方案。然而,从像素观测学习SFs的传统方法常导致表示崩溃,即特征退化而无法捕捉数据有意义变化。近期方法虽可避免表示崩溃,但往往涉及复杂损失函数与多阶段学习,降低效率。本文提出一种新颖、简洁的方法,直接从像素学习SFs。该方法结合时序差分(TD)损失与奖励预测损失,共同捕捉SFs的基本数学定义。实验表明,该方法在2D(Minigrid)、3D(Miniworld)迷宫及Mujoco环境中,无论单任务还是持续学习场景,均达到或超过现有技术性能。同时,本方法高效,可在更短时间内达成更高表现。工作提供了一种无需预训练、从像素直接学习成功续特征的新途径。
原文摘要 · Abstract (English)
In Deep Reinforcement Learning (RL), it is a challenge to learn representations that do not exhibit catastrophic forgetting or interference in non-stationary environments. Successor Features (SFs) offer a potential solution to this challenge. However, canonical techniques for learning SFs from pixel-level observations often lead to representation collapse, wherein representations degenerate and fail to capture meaningful variations in the data. More recent methods for learning SFs can avoid representation collapse, but they often involve complex losses and multiple learning phases, reducing their efficiency. We introduce a novel, simple method for learning SFs directly from pixels. Our approach uses a combination of a Temporal-difference (TD) loss and a reward prediction loss, which together capture the basic mathematical definition of SFs. We show that our approach matches or outperforms existing SF learning techniques in both 2D (Minigrid), 3D (Miniworld) mazes and Mujoco, for both single and continual learning scenarios. As well, our technique is efficient, and can reach higher levels of performance in less time than other approaches. Our work provides a new, streamlined technique for learning SFs directly from pixel observations, with no pretraining required.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。