提出CompoFormer模型,实现离线强化学习中持续任务的稳定与快速适应。
Continual Task Learning through Adaptive Policy Self-Composition
- 通过元策略网络自适应组合历史策略,提升知识复用效率。
- 在长序列任务中表现优于传统方法,避免灾难性遗忘。
- 适用于需要长期学习、数据不连续的智能体场景。
让通用智能体从离线轨迹中持续学习一系列任务,是长期运行智能体的自然需求,但现有离线强化学习算法仍面临重大挑战。具体而言,智能体需在新任务上快速适应(可塑性),同时保留先前任务的知识(稳定性)。然而,对此类场景的系统分析较少,传统持续学习方法在离线持续强化学习(CORL)中的有效性尚不明确。本研究构建了离线持续世界基准,发现传统方法易因CORL特有的分布偏移导致灾难性遗忘。为此,我们提出CompoFormer——一种基于结构的持续变压器模型,通过元策略网络自适应组合先前策略。面对新任务时,该模型利用语义关联选择性融合相关历史策略与新训练参数,从而增强知识共享并加速学习。实验表明,CompoFormer在长任务序列中显著优于传统持续学习方法,展现出良好的可塑性与稳定性平衡。
原文摘要 · Abstract (English)
Training a generalizable agent to continually learn a sequence of tasks from offline trajectories is a natural requirement for long-lived agents, yet remains a significant challenge for current offline reinforcement learning (RL) algorithms. Specifically, an agent must be able to rapidly adapt to new tasks using newly collected trajectories (plasticity), while retaining knowledge from previously learned tasks (stability). However, systematic analyses of this setting are scarce, and it remains unclear whether conventional continual learning (CL) methods are effective in continual offline RL (CORL) scenarios. In this study, we develop the Offline Continual World benchmark and demonstrate that traditional CL methods struggle with catastrophic forgetting, primarily due to the unique distribution shifts inherent to CORL scenarios. To address this challenge, we introduce CompoFormer, a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network. Upon encountering a new task, CompoFormer leverages semantic correlations to selectively integrate relevant prior policies alongside newly trained parameters, thereby enhancing knowledge sharing and accelerating the learning process. Our experiments reveal that CompoFormer outperforms conventional CL methods, particularly in longer task sequences, showcasing a promising balance between plasticity and stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。